Deep Q-Networks (DQN)
2013: Mnih trained one Q-learning network on raw pixels, beat every classical RL agent on seven Atari games. 2015: extended to 49 games, published in Nature, sparked the deep-RL era. DQN is Q-learning plus three tricks that make function approximation stable. Tabular Q-learning needs a separate Q-value for every (state, action) pair. A chess board has 10⁴³ states. An Atari frame is 210×160×3 = 100,800 features. Tabular RL dies at thousands of states, let alone billions. The fix is obvious in hindsight: replace the Q-table with a neural network, Q(s, a; θ). But obvious-in-hindsight took decades. Naive function approximation with Q-learning diverges under the "deadly triad" — function approximation + bootstrapping + off-policy learning. Mnih et al. (2013, 2015) identified three engineering tricks that stabilize learning: Experience replay decorrelates transitions. Target network freezes the bootstrap target. Reward clipping normalizes gradient magnitudes. DQN on Atari was the first time a single architecture with a single hyperparameter set solved dozens of control problems from raw pixels. Everything "deep-RL" built since — DDQN, Rainbow, Dueling, Distributional, R2D2, Agent57 — is stacked on top of this three-trick base. DQN training loop: env, replay buffer, online net, target net, Bellman TD loss The objective. DQN minimizes the one-step TD loss on a neural Q-function: L(θ) = E{(s,a,r,s')D} [ (r + γ max{a'} Q(s', a'; θ^-)…
Deep Q-Networks (DQN): 2013: Mnih trained one Q-learning network on raw pixels, beat every classical RL agent on seven Atari games. 2015: extended to 49…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.