How would you read a JSON file in PySpark?

Instruction: Describe the steps and the PySpark SQL function you would use to read a JSON file.

Context: This question is designed to evaluate the candidate's practical skills in handling and processing data in PySpark. It tests their familiarity with PySpark's built-in functions for reading different data formats and their ability to apply this knowledge to ingest data from a JSON file.

Updated

Use spark.read.json(path). The default file format is JSON Lines: each line contains a complete JSON object. Set multiLine=True for a whole JSON document that spans lines. These are different input layouts, not merely different file sizes.

With an existing classic local SparkSession named spark, this complete fixture reads a temporary JSON Lines file:

from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as directory:
    path = Path(directory) / "people.json"
    path.write_text(
        '{"name":"Ada","age":31}\n'
        '{"name":"Bo","age":null}\n', encoding="utf-8"
    )
    people = (spark.read.schema("name string, age int")
              .option("mode", "FAILFAST").json(str(path)))
    result = people.orderBy("name").collect()
    assert [(r.name, r.age) for r in result] == [("Ada", 31), ("Bo", None)]

The action occurs while the temporary file exists. On a cluster, use storage accessible to the executors; a driver-local temporary path is not a shared-storage design.

An explicit schema avoids inference work and stabilizes intended types. It does not enforce required business fields, uniqueness or ranges. Validate those after parsing. FAILFAST reports malformed records when they are read; choosing PERMISSIVE requires an explicit corrupt-record field and a reject-handling policy. Do not silently drop bad records or confuse successful parsing with complete validation. For a multi-line document, configure .option("multiLine", True) on the reader instead. JSON formats and parser options

Related Questions