Demonstrate how to use the PySpark filter function

Instruction: Write a PySpark code snippet that uses the filter function to select rows from a DataFrame where the values in the 'age' column are greater than 30.

Context: This question evaluates the candidate's proficiency with PySpark's DataFrame API, specifically their ability to apply transformations to select data based on specific criteria. It also tests their understanding of PySpark's functional programming paradigm.

Updated

filter returns a new DataFrame containing rows for which its Boolean condition is true. This complete local example includes the threshold and null cases:

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.master("local[2]").appName("FilterExample").getOrCreate()
try:
    df = spark.createDataFrame(
        [("Ava", 30), ("Bo", 31), ("Cy", None), ("Dee", 45)],
        "name string, age int",
    )
    filtered_df = df.filter(F.col("age") > 30)
    filtered_df.orderBy("name").show()
finally:
    spark.stop()

The output contains Bo, 31 and Dee, 45. Age 30 is excluded because the comparison is strict. A null age yields an unknown comparison rather than true, so that row is excluded too. where is an alias for filter; a literal SQL condition such as df.filter("age > 30") also works. Filter API

The transformation is lazy; the action show() triggers execution. orderBy makes this demonstration's displayed order deterministic. The original DataFrame still contains all four records. If unknown ages should be included, express that policy explicitly with F.col("age").isNull() | (F.col("age") > 30). Parenthesize comparisons and use & or | for compound column expressions, rather than Python and or or. Null semantics

The example owns a local Spark session and stops it. In a managed notebook or shared application, reuse the supplied session and follow that environment's lifecycle instead.

Related Questions