Explain Trust Region Policy Optimization (TRPO) and its advantages over other policy gradient methods.

Example Answer

TRPO updates a policy while limiting how far its action distribution moves from the previous policy. It optimizes a surrogate objective subject to a KL-divergence constraint, using an approximate natural-gradient direction and a line search in the practical algorithm.

That can make updates more controlled than an unconstrained policy-gradient step. The important qualification is that the original monotonic-improvement result relies on a theoretical procedure and assumptions. Practical TRPO uses approximations, sampled estimates, and an average KL constraint, so I would not claim every update is guaranteed to improve actual return.

The trade-off is computational and implementation complexity: Fisher-vector products, conjugate-gradient calculations, and line-search checks add work. PPO offers a simpler first-order alternative, although it has its own limits. For either method, I would monitor held-out return, actual KL, and update acceptance across multiple runs. A trust region controls policy change; it does not certify that a robot's actions are physically safe.

Concrete Check

If a proposed step exceeds the chosen KL limit or fails the surrogate-improvement check, a practical TRPO line search should reduce or reject it. That check still depends on the sampled batch.

Original TRPO paper