Scribd Logo
Upload

Welcome to Scribd!

  • ATELIER RDD - Esprit Correction

    Uploaded by

    infode prode
    16 views1 page
    The document shows Spark code for analyzing purchase and log files. It loads a purchases text file into an RDD with the default number of partitions, then repartitions it to 4 partitions. It transforms the RDD to extract customer IDs and amounts, aggregates by customer to sum the amounts, and collects the results. It also loads an error log file, filters for lines containing "ERROR", counts them, further filters for lines containing "php", collects the results, and inspects the RDD debug string.

    Original Description:

    Copyright

    © © All Rights Reserved

    Available Formats

    DOCX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document shows Spark code for analyzing purchase and log files. It loads a purchases text file into an RDD with the default number of partitions, then repartitions it to 4 partitions. It transforms the RDD to extract customer IDs and amounts, aggregates by customer to sum the amounts, and collects the results. It also loads an error log file, filters for lines containing "ERROR", counts them, further filters for lines containing "php", collects the results, and inspects the RDD debug string.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as docx, pdf, or txt
    16 views1 page

    ATELIER RDD - Esprit Correction

    Uploaded by

    infode prode
    The document shows Spark code for analyzing purchase and log files. It loads a purchases text file into an RDD with the default number of partitions, then repartitions it to 4 partitions. It transforms the RDD to extract customer IDs and amounts, aggregates by customer to sum the amounts, and collects the results. It also loads an error log file, filters for lines containing "ERROR", counts them, further filters for lines containing "php", collects the results, and inspects the RDD debug string.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as docx, pdf, or txt
    of 1

    ATELIER RDD

    1—

    rdd = sc.textFile("atelier1/purchases.txt")

    rdd.getNumPartitions()

    rdd2_coalesce = rdd.coalesce(4)

    rdd2_coalesce.getNumPartitions()

    tab = rdd.map(lambda x : x.split("\t"))

    tab.map(lambda n: (str(n[2]), float(n[4]) ) ).reduceByKey(lambda v1,v2: v1 + v2).collect()

    2---

    logFile = sc.textFile("atelier1/error_log.txt")

    error = logFile.filter(lambda line: "ERROR" in line)

    error.count()

    error.filter(lambda line: "php" in line).collect()

    error.toDebugString()

    You might also like