What strategies would you use to debug a PySpark application that is running slowly?
Instruction: Outline your approach to identify and resolve performance bottlenecks in a PySpark application.
Updated
Start with a reproducible input, expected output and latency or completion-time baseline. Record the Spark/runtime versions and whether the slowdown is new, intermittent or proportional to input growth.
Use the SQL plan and Spark UI to locate the expensive stage. Compare task-duration distributions, records, shuffle bytes, spill, garbage-collection time, failed/retried tasks and source/sink throughput. A few slow tasks suggest a different problem from every task being slow. Check driver-side planning, file listing and accidental collection as well as executor work.
Choose a change that addresses the evidence:
- For excessive data movement, remove unneeded columns, apply valid filters early, inspect join cardinality and compare join strategies. An unnecessary repartition adds a shuffle.
- For a hot key, ordinary hash repartitioning by that same key does not split it. Consider supported adaptive skew handling or a correctness-preserving decomposition/salting design.
- For repeated expensive computation, persist only a reusable result that fits the resource budget, materialize it deliberately and unpersist it afterward.
- For Python overhead, compare an equivalent built-in expression; broadcasting a local Python object differs from a DataFrame broadcast join.
- For resource pressure, inspect heap, Python memory, disk and concurrency before changing executor settings. More cores can increase simultaneous memory demand.
Change one hypothesis at a time and compare repeated equivalent runs, including cold/warm-cache conditions, output row multiplicity and failures. Keep the measured configuration and rollback path. A smaller sample can reproduce logic errors, but may hide skew, file-count or scale effects. Refactoring for readability is useful; it is not evidence that a performance bottleneck is fixed.