Describe the SARSA algorithm and how it differs from Q-learning.

Instruction: Provide an overview of the SARSA algorithm, including its key characteristics and how it compares to Q-learning.

Context: The question evaluates the candidate's knowledge of specific reinforcement learning algorithms and their ability to compare and contrast different approaches.

Updated

Example Answer

SARSA is an on-policy temporal-difference control method. Its update uses the sequence state, action, reward, next state, and the next action actually selected by the current policy. It therefore learns values for the policy's behavior, including its exploration.

Q-learning uses the maximum estimated next-action value instead. It targets greedy behavior even if the agent will actually take an exploratory action. The difference is the continuation target, not whether either method looks only at immediate reward.

In a cliff-walking example, SARSA can prefer a route that accounts for the risk of exploratory moves near the cliff. That does not make SARSA an inherently safe algorithm or Q-learning inherently reckless. Their behavior depends on exploration, rewards, and the environment. I would evaluate unsafe outcomes explicitly rather than treating the algorithm name as a safety guarantee.

Worked Example

For reward 1 and γ = 0.9, suppose the next-state action values are [4, 2], and the policy selects the second action. SARSA's target is 1 + 0.9 × 2 = 2.8; Q-learning's target is 1 + 0.9 × 4 = 4.6. At a terminal state, both targets are just 1.

David Silver: model-free control

Related Questions