Optimizing join operations in PySpark for large-scale data processing

Verify join correctness first: intended grain, key types, null treatment, duplicate keys and whether a many-to-many result is expected. A fast join that multiplies revenue is still wrong.

Project unused columns away and apply valid filters before the join. Inspect estimated statistics and the executed physical plan. Spark may choose broadcast, sort-merge or another supported strategy; hints do not guarantee a strategy for every join type. Broadcast only a side whose expanded representation fits safely in each executor alongside other work. Join strategies

repartition(n, "key") can establish hash partitioning, but adds its own shuffle. Compatible existing partitioning may be reusable; blindly repartitioning both inputs does not guarantee less total work or physical co-location. coalesce(n) only changes partition count and cannot partition by join-key columns. Check whether the actual plan introduces exchanges. Coalesce behavior

Measure key skew and task sizes. Adaptive Query Execution can mitigate some skew. If salting is necessary, design the full matching rule: distribute hot-key rows on one side and replicate the corresponding rows on the other across the same salt values, then remove the salt. Independently assigning random salts to both sides loses valid matches. Account for replication cost, deterministic retries and duplicate semantics; validate the salted result against an unsalted reference on a bounded fixture.

Choose a semi or anti join only when the business question is existence or nonexistence; changing join type is not merely a performance switch. Benchmark identical outputs using shuffle bytes, spills, task skew, executor memory and elapsed time. Retain a change only when it improves the relevant workload without changing row multiplicity or dropping unmatched rows that the contract requires.