Deep Q-Learning

Deep Q-Learning is an extension of Q-Learning that uses a neural network to approximate the Q-Table. This allows reinforcement learning to scale to environments with large or continuous state-action spaces.

Why Deep Q-Learning?

Deep Q-Learning addresses the limitations of traditional Q-Learning:

• Handles high-dimensional state spaces (e.g., images).
• Avoids the need to store large Q-Tables in memory.
• Generalizes across states using neural networks.

The DQN Algorithm

Deep Q-Learning uses a neural network called the Q-Network to approximate the Q-values. The main steps are:

1. Initialize the Q-Network with random weights.
2. Interact with the environment to collect experience tuples (state, action, reward, next_state).
3. Update the Q-Network using the Bellman equation:

Q(s, a) ≈ r + γ max(Q(s', a'))

Building the Q-Network

The Q-Network is a simple feedforward neural network that takes the current state as input and outputs Q-values for all possible actions:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# Define the Q-Network
q_network = Sequential([
    Dense(24, activation='relu', input_shape=(state_space_size,)),
    Dense(24, activation='relu'),
    Dense(action_space_size, activation='linear')
])

q_network.compile(optimizer='adam', loss='mse')

Experience Replay

Deep Q-Learning uses a technique called experience replay to improve learning:

• Store experiences (state, action, reward, next_state) in a memory buffer.
• Randomly sample batches of experiences to train the Q-Network.

from collections import deque

# Initialize replay memory
memory = deque(maxlen=2000)

# Add experience to memory
memory.append((state, action, reward, next_state, done))

Training the Q-Network

We train the Q-Network using the experiences from the memory buffer:

import numpy as np

# Sample a batch of experiences
batch_size = 32
batch = random.sample(memory, batch_size)

# Train the Q-Network
for state, action, reward, next_state, done in batch:
    target = reward
    if not done:
        target += gamma * np.max(q_network.predict(next_state[np.newaxis, ...]))
    q_values = q_network.predict(state[np.newaxis, ...])
    q_values[0][action] = target
    q_network.fit(state[np.newaxis, ...], q_values, epochs=1, verbose=0)

Target Networks

To stabilize training, DQN uses a target network:

• Maintain a separate network for calculating target Q-values.
• Periodically copy weights from the Q-Network to the target network.

Advantages of Deep Q-Learning

Deep Q-Learning offers several benefits:

• Handles high-dimensional state spaces, like images.
• Generalizes across states using neural networks.
• Can solve complex tasks like Atari games and robotic control.

Summary and Next Steps

In this lesson, we:

• Explored the basics of Deep Q-Learning.
• Built a Q-Network to approximate Q-values.
• Discussed experience replay and target networks for stable training.

Next, we will explore OpenAI Gym and apply reinforcement learning in simulated environments.