Scribd Logo
Upload

Welcome to Scribd!

  • 1K views

    DataFrame Operations Using A Json File

    Uploaded by

    Arpita Das
    This Python code uses Spark SQL to read employee data from a JSON file into a DataFrame. It then filters the DataFrame to only rows where the stream is "JAVA" and writes the filtered DataFrame to a new Parquet file. It first reads the JSON, displays the DataFrame, coalesces and writes it to a Parquet file. Then it reads the Parquet, filters for "JAVA" stream, displays and writes the filtered DataFrame to a new Parquet file.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content

    You might also like

    DataFrame Operations Using A Json File

    Uploaded by

    Arpita Das
    1K views1 page
    This Python code uses Spark SQL to read employee data from a JSON file into a DataFrame. It then filters the DataFrame to only rows where the stream is "JAVA" and writes the filtered DataFrame to a new Parquet file. It first reads the JSON, displays the DataFrame, coalesces and writes it to a Parquet file. Then it reads the Parquet, filters for "JAVA" stream, displays and writes the filtered DataFrame to a new Parquet file.

    Original Description:

    Original Title

    DataFrame Operations Using a Json File

    Copyright

    © © All Rights Reserved

    Available Formats

    TXT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This Python code uses Spark SQL to read employee data from a JSON file into a DataFrame. It then filters the DataFrame to only rows where the stream is "JAVA" and writes the filtered DataFrame to a new Parquet file. It first reads the JSON, displays the DataFrame, coalesces and writes it to a Parquet file. Then it reads the Parquet, filters for "JAVA" stream, displays and writes the filtered DataFrame to a new Parquet file.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as txt, pdf, or txt
    1K views1 page

    DataFrame Operations Using A Json File

    Uploaded by

    Arpita Das
    This Python code uses Spark SQL to read employee data from a JSON file into a DataFrame. It then filters the DataFrame to only rows where the stream is "JAVA" and writes the filtered DataFrame to a new Parquet file. It first reads the JSON, displays the DataFrame, coalesces and writes it to a Parquet file. Then it reads the Parquet, filters for "JAVA" stream, displays and writes the filtered DataFrame to a new Parquet file.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as txt, pdf, or txt
    of 1

    #Put your code here

    from pyspark.sql import SparkSession


    spark = SparkSession \
    .builder \
    .appName("Data Frame EMPLOYEE") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()
    df = spark.read.json("emp.json")
    df.show()
    df.coalesce(1).write.parquet("Employees")
    pf = spark.read.parquet("Employees")
    dfNew = pf.filter(pf.stream=='JAVA')
    dfNew.show()
    dfNew.coalesce(1).write.parquet("JavaEmployees")

    You might also like