Discuss the role and importance of batch normalization in deep learning.
Instruction: Explain what batch normalization is and how it benefits model training.
Updated
Example Answer
Batch normalization normalizes each feature or channel using minibatch statistics, then applies a learned scale and offset. It often improves optimization and permits larger learning rates, although the benefit depends on the architecture and training setup.
Internal covariate shift was the original motivation. I would not present it as a settled explanation of why BatchNorm works; later research found that improved optimization geometry can be important.
I pay particular attention to training and inference behavior. With PyTorch's default running-statistics configuration, training uses batch statistics and updates running estimates; evaluation uses those stored estimates. A small or unrepresentative batch can produce noisy statistics, and deployment data can differ from the running estimates. LayerNorm or GroupNorm may be better fits when batch statistics are unreliable. Placement relative to activation is an architecture choice, not a universal rule.
Concrete Check
Before validation, call model.eval(). Disabling gradients alone does not switch BatchNorm into evaluation behavior. Conversely, evaluation mode does not itself disable gradient tracking.