How do you cache a DataFrame in PySpark, and why would you do it?

Instruction: Explain the process and reasoning behind caching a DataFrame in PySpark.

Context: This question is designed to test the candidate's knowledge of caching or persisting data in PySpark to optimize the performance of data processing applications.

Updated

Official answer

Use df.cache() when several actions reuse an expensive DataFrame computation. Caching is lazy: an action computes the partitions it needs and makes them available for reuse. Calling cache() alone does not eagerly materialize the whole dataset.

For PySpark DataFrames, the default has been MEMORY_AND_DISK_DESER since Spark 3.0. It is incorrect to equate DataFrame cache() with MEMORY_ONLY; do not transfer an RDD default to the DataFrame API. persist(storageLevel) allows an explicit supported storage level.

For example, after obtaining a SparkSession named spark:

from pyspark.sql import functions as F

shared = spark.range(100).where("id % 2 = 0").cache()
try:
    count = shared.count()  # An action materializes the needed partitions.
    total = shared.agg(F.sum("id").alias("total")).first()["total"]
    assert (count, total) == (50, 2450)
finally:
    shared.unpersist(blocking=True)

The tiny example demonstrates reuse and cleanup, not a performance improvement. In a real job, cache a useful shared result after appropriate filtering/projection, inspect the storage view and query plan, and compare total job time including materialization. Memory pressure, disk spill and cache management can outweigh saved recomputation. A single-use DataFrame often does not benefit.

Unpersist when the result is no longer needed. A cache is rebuildable execution state, not a durable output, backup or guarantee against executor loss.

Sources

PySpark DataFrame.cache; Spark SQL performance tuning.

Related Questions