Scribd Logo
Upload

Welcome to Scribd!

  • Spark Best Practices

    Uploaded by

    Rahul yadav
    10 views2 pages
    Spark dataframes should be used instead of Spark SQL for filtering data as dataframes provide better performance and are easier to use than SQL queries. The dataframe API allows filtering by specifying the column and values to filter on directly in the dataframe whereas Spark SQL requires writing out the full SQL query string.

    Original Description:

    Spark Best Practices

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    Spark dataframes should be used instead of Spark SQL for filtering data as dataframes provide better performance and are easier to use than SQL queries. The dataframe API allows filtering by specifying the column and values to filter on directly in the dataframe whereas Spark SQL requires writing out the full SQL query string.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    10 views2 pages

    Spark Best Practices

    Uploaded by

    Rahul yadav
    Spark dataframes should be used instead of Spark SQL for filtering data as dataframes provide better performance and are easier to use than SQL queries. The dataframe API allows filtering by specifying the column and values to filter on directly in the dataframe whereas Spark SQL requires writing out the full SQL query string.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as pdf or txt
    of 2

    Spark Best Practices

    Use Spark dataframe APIs instead of


    Spark SQL

    sqlContext.table(“table_schema.table_name”).filte
    r(col(“prod_cd ”)in (‘’,’’))

    spark.sql(“select * from
    table_schema.table_name where
    prod_cd in (‘’,’’,’’)”)

    You might also like