Can you discuss the impact of reward shaping in reinforcement learning models?

Instruction: Explain reward shaping and potential-based shaping. For discount gamma=0.9, an agent can finish immediately for reward 1 or remain forever in a nonterminal loop with original reward 0. A heuristic adds +0.2 to every loop transition. Compare original and shaped returns and state boundary conditions for policy preservation.

Context: This question assesses the candidate's insight into techniques used to modify the learning process in reinforcement learning for better performance or faster convergence.

Updated

Worked answer

Originally, finishing yields return 1 and looping forever yields 0, so finishing is preferred. With the heuristic bonus, looping yields 0.2+0.9×0.2+0.9²×0.2+…=0.2/(1−0.9)=2, while immediate finishing still yields 1. The agent now prefers the loop: faster learning of the shaped objective would not mean better performance on the original task.

Potential-based shaping adds F(s,s′)=γΦ(s′)−Φ(s) using the same discount as the task. Its discounted sum telescopes: over T transitions it is −Φ(s0)+γ^T Φ(sT). With bounded potential in a discounted infinite horizon, the final term vanishes; for a finite episode, terminal potential must be handled so that it does not introduce a policy-dependent terminal bonus. Under those conditions the change is a state-dependent constant for a fixed start, preserving policy preferences.

Document the horizon, discount and terminal convention. Test loops, early termination and reward hacking, and evaluate with the original reward. An arbitrary progress bonus has no such policy-preservation guarantee. Shaping can affect learning efficiency without proving robustness or task success.

Reference: Policy invariance under reward transformations.

Related Questions