Explain how you would optimize a PySpark job for processing a multi-terabyte dataset on a cluster with limited memory.

Start with the executed plan and Spark UI: identify whether the limiting task is scanning, shuffling, skew, Python execution, garbage collection, spill or output. Use representative data and keep correctness checks alongside timing.

Reduce work before increasing memory. Read only required columns, use partition pruning and supported predicate pushdown, and prefer built-in DataFrame expressions over per-row Python UDFs when they express the operation. Avoid moving a multi-terabyte result into driver memory with collect() or toPandas().

Size shuffle partitions so individual tasks fit the available working memory while keeping useful parallelism. repartition() itself introduces a shuffle; increasing its count does not inherently reduce total bytes shuffled. Too few partitions or a hot key can cause oversized tasks, while excessive tiny partitions add overhead. Inspect Adaptive Query Execution and skew handling before forcing an additional exchange. SQL tuning

Persist only reused intermediates, select an appropriate memory/disk storage level and unpersist after reuse. Broadcasting a genuinely small join side can help, but the expanded representation must fit each executor alongside concurrent work. Broadcasting a large table can make the problem worse.

Separate executor JVM heap from Python worker memory and other overhead. PySpark Python-object serialization is not simply a choice between Marshal and Kryo; Kryo primarily concerns relevant JVM-side serialization and does not replace every Python serialization path. Python object serialization

Within the JVM, spark.memory.fraction sets the unified execution/storage region, and spark.memory.storageFraction sets its protected storage portion; it is not an internal-overhead reserve. Change those fractions only with evidence. Tune executor size and concurrent tasks within the actual container budget, and allow sufficient local disk for expected spills. Memory management

Compare repeated runs for output equivalence, task-size distribution, peak memory, spill, runtime and cost. Report the measured improvement for that workload rather than guaranteeing a universal speedup.