Can you explain what Q-learning is?

Instruction: Provide a brief overview of Q-learning.

Context: This question tests the candidate's understanding of Q-learning, a model-free Reinforcement Learning algorithm used to inform an agent about the utility of its actions.

Updated

Example Answer

Q-learning is a model-free, off-policy algorithm that estimates the expected discounted return from taking an action in a state and then following a greedy policy. After a transition, it moves the current estimate toward the observed reward plus the best estimated value at the next state.

The behavior policy can explore, while the update targets greedy future action selection. For a genuinely terminal transition, the target is just the reward. An external time limit is different: it can require a reset while still leaving a continuation value to bootstrap from.

In the tabular setting, convergence requires assumptions such as sufficient state-action coverage and suitable diminishing learning rates. Replacing the table with a neural network does not automatically preserve that guarantee. I would verify the target and terminal handling on a tiny environment before tuning a larger agent.

Worked Example

q = 2.0
reward, gamma, alpha = 1.0, 0.9, 0.5
next_values = [3.0, 4.0]
terminated = False
target = reward if terminated else reward + gamma * max(next_values)
q += alpha * (target - q)
print(round(q, 2))  # 3.3; a terminal transition would give 1.5

Q-learning paper · Gymnasium termination versus truncation

Related Questions