How can you improve the performance of a PySpark application?

Start with a reproducible workload and a correct expected result. Use the Spark UI and query plan to identify whether time is spent scanning, shuffling, spilling, running Python code, waiting on external I/O or processing a few skewed tasks.

Three useful optimizations address different causes:

1. Reduce repeated work. Persist an expensive intermediate result only when it is reused enough to justify storage and materialization costs. Choose a storage level, inspect actual reuse and call unpersist when finished. Caching a single-use result can make the job worse.
2. Reduce movement and oversized tasks. Filter and project before expensive operations when semantics permit. Tune shuffle partitions using actual task sizes and inspect adaptive execution. repartition(n, "key") redistributes a DataFrame by a key; write.partitionBy("key") lays out output files and does not promise permanent placement on the same executor. A hot key remains a hot key under ordinary hash partitioning.
3. Choose a suitable join. A small dimension may fit a broadcast join using the broadcast function from pyspark.sql.functions. Check its materialized size, executor memory and the final plan. This is different from SparkContext.broadcast, which distributes a local read-only Python value; it is not a way to broadcast an RDD object.

Prefer built-in DataFrame expressions when they express the operation, use an appropriate columnar storage layout, and validate join cardinality. A large accidental many-to-many result cannot be repaired by adding memory alone. Measure the same input/output semantics before and after each change, including failures, memory and long-tail task duration. These are hypotheses to test, not universal speedups. SQL performance tuning, Broadcast variables