Discuss the importance of normalization techniques in reinforcement learning models.

Example Answer

I distinguish observation normalization, reward scaling, advantage normalization, and gradient clipping because they affect different parts of an RL system. Observation normalization can improve conditioning, but I keep its statistics consistent between training and evaluation and save them with the policy.

Advantage normalization can make optimization less sensitive to batch scale. Gradient norm clipping bounds the gradient passed to the optimizer; the actual parameter change also depends on the learning rate and optimizer state. Neither guarantees stable learning. Reward transformations deserve particular care: clipping or centering rewards can change which policy is best, rather than simply making the same objective easier to optimize.

For example, clipping rewards of 1 and 100 to the same value removes the preference between those outcomes. If I want extra guidance while preserving an MDP's optimal policies, potential-based shaping has a specific form and assumptions; an arbitrary bonus does not share that guarantee. I would always report evaluation returns in the original task units and check that a preprocessing change has not silently redefined success.

Batch normalization acts on network activations and is separate from input normalization. When the implementation tracks running statistics, preserve them and choose training/evaluation mode deliberately; RL's changing observation distribution can make stale statistics misleading. Validate this choice rather than assuming it stabilizes every agent.

Concrete Example

For plain scalar SGD with a clipped gradient of 1, learning rates of 0.1 and 2 produce update magnitudes of 0.1 and 2. The clipping threshold alone is not the step-size bound.

Potential-based shaping uses r' = r + γΦ(next_state) − Φ(state). Its invariance conditions include consistent discounting and appropriate terminal handling. For episodic tasks, setting terminal potentials to zero avoids an action-dependent final-potential bonus.

Policy-invariant reward shaping · Gradient clipping implementation · Batch normalization running statistics