How do you deal with null values in DataFrame in PySpark?

Instruction: Explain the methods available in PySpark for handling missing or null values in a DataFrame.

Context: This question tests the candidate's ability to manage and manipulate data that may contain null values, which is a common scenario in real-world data processing tasks.

Updated

Choose a missing-value policy from the field's meaning. SQL NULL differs from floating-point NaN, an empty string or a sentinel such as -1. Ordinary comparisons with null yield unknown, so a filter keeps only rows whose predicate is true.

For an existing SparkSession named spark:

from pyspark.sql import functions as F

df = spark.createDataFrame([
    (1, None, None), (2, float("nan"), ""), (3, 0.0, "A")
], "id int, value double, category string")
assert df.filter(F.col("value").isNull()).count() == 1
assert df.filter(F.isnan("value")).count() == 1
kept = df.na.drop(subset=["value"])
assert [r.id for r in kept.collect()] == [3]
filled = df.na.fill({"value": 0.0, "category": "Unknown"})
assert [(r.value, r.category) for r in filled.orderBy("id").collect()] == [
    (0.0, "Unknown"), (0.0, ""), (0.0, "A")]

For these numeric columns, dropna/fillna handle NaN as well as null; the empty string is unchanged. fillna takes replacement values: it does not compute a mean or median just because a statistic is desired. Compute or fit that statistic separately, using training data only for predictive modeling. Type-incompatible replacements may be ignored, so check the result. Fill behavior

Keep the original field or a missingness flag when replacement could be confused with a genuine zero or category. Use coalesce/when for explicit per-field rules and the numeric ML Imputer for supported fitted strategies. Neither exploratory analysis nor a large dataset makes selection bias harmless; report exclusions and test all-missing, empty and invalid-value cases.

Related Questions