Scribd Logo
Upload

Welcome to Scribd!

  • Notes

    Uploaded by

    cloudtraining2023
    9 views1 page
    This document demonstrates using mapValues and reduceByKey to count word frequencies in a dataset. It first maps each value to a tuple containing the original value and a count of 1. It then reduces by key, summing the first element of each tuple and counting the number of tuples for each key. This provides the word counts per unique key in the output RDD.

    Original Description:

    Copyright

    © © All Rights Reserved

    Available Formats

    TXT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This document demonstrates using mapValues and reduceByKey to count word frequencies in a dataset. It first maps each value to a tuple containing the original value and a count of 1. It then reduces by key, summing the first element of each tuple and counting the number of tuples for each key. This provides the word counts per unique key in the output RDD.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    9 views1 page

    Notes

    Uploaded by

    cloudtraining2023
    This document demonstrates using mapValues and reduceByKey to count word frequencies in a dataset. It first maps each value to a tuple containing the original value and a count of 1. It then reduces by key, summing the first element of each tuple and counting the number of tuples for each key. This provides the word counts per unique key in the output RDD.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 1

    RDD ==>

    panda,0
    pink,3
    pirate,3
    panda,1
    pink,4

    operation ==>

    rdd.mapValues(lambda x:(x,1)).reduceBykeys(lambda x,y:(x[0]+y[0],x[1]+y[1]))

    output:
    panda, (1,2)
    pink,(7,2)
    pirate,(3,1)

    explaination :
    mapValues(lambda x:(x,1)) outputs a new rdd as -
    {(panda,(0,1)),
    (pink,(3,1)),
    (pirate,(3,1)),
    (panda,(1,1)),
    (pink,(4,1))
    }

    reduceBykeys(lambda x,y:(x[0]+y[0],x[1]+y[1])) cummulatively adds values together


    for same keys as -
    panda, ((0+1),(1+1))
    pink,((3+4),(1+1))
    pirate,(3,1)

    You might also like