Scribd Logo
Upload

Welcome to Scribd!

  • Spark Shuffle

    Uploaded by

    dd
    5 views102 pages
    The document describes how Spark processes data distributed across a cluster using operations like map, shuffle, groupByKey and reduceByKey. First, Spark maps the input data to (key, value) pairs. It then shuffles the data across the network to group values with the same key together on individual nodes. Next, groupByKey is used to fully group the values under each key. Finally, reduceByKey locally aggregates the values for each key.

    Original Description:

    Original Title

    spark_shuffle

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document describes how Spark processes data distributed across a cluster using operations like map, shuffle, groupByKey and reduceByKey. First, Spark maps the input data to (key, value) pairs. It then shuffles the data across the network to group values with the same key together on individual nodes. Next, groupByKey is used to fully group the values under each key. Finally, reduceByKey locally aggregates the values for each key.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    5 views102 pages

    Spark Shuffle

    Uploaded by

    dd
    The document describes how Spark processes data distributed across a cluster using operations like map, shuffle, groupByKey and reduceByKey. First, Spark maps the input data to (key, value) pairs. It then shuffles the data across the network to group values with the same key together on individual nodes. Next, groupByKey is used to fully group the values under each key. Finally, reduceByKey locally aggregates the values for each key.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 102

    !

    !
    !
    !

    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    Workers

    SHUFFLE
    What might this look like on the cluster? “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)
    Workers

    SHUFFLE
    What might this look like on the cluster? “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)
    Workers

    SHUFFLE
    What might this look like on the cluster? “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)
    Workers

    SHUFFLE
    What might this look like on the cluster? “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)

    groupByKey
    (100, [22.25, 12.40, 31.60]) (200, [8.20]) (300, [42.10, 16.20])
    Workers

    What might this look like on the cluster?

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)

    SHUFFLE
    “Shuffles” data
    across network groupByKey
    (100, [22.25, 12.40, 31.60]) (200, [8.20]) (300, [42.10, 16.20])
    Shared Memory Distributed

    Seconds Days
    L1 cache reference..........0.5s
    Roundtrip within
    L2 cache reference............7s same datacenter.........5.8 days
    Mutex lock/unlock............25s

    Minutes Years
    Main memory reference.....1m 40s Send packet
    CA->Netherlands->CA....4.8 years
    Workers

    SHUFFLE
    What might this look like on the cluster? “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)
    Workers

    SHUFFLE
    What might this look like on the cluster? “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)
    Workers

    What might this look like on the cluster? SHUFFLE


    “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, (1, 22.25)) (100, (1, 12.40)) (300, (1, 42.10))
    (100, (1, 31.60)) (200, (1, 8.20)) (300, (1, 16.20))

    (300, [42.10, 16.20])


    Workers

    What might this look like on the cluster? SHUFFLE


    “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, (1, 12.40)) reduce
    (100, (2, 53.85))
    (200, (1, 8.20))
    (300, (2, 58.30))
    on the
    mapper
    side
    first!
    reduceByKey

    (300, [42.10, 16.20])

    (100, [22.25, 12.40, 31.60]) (200, [8.20]) (300, [42.10, 16.20])


    Workers

    What might this look like on the cluster? SHUFFLE


    “Shuffles” data
    across network

    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)


    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    map
    (100, (1, 12.40))
    (100, (2, 53.85)) (300, (2, 58.30))
    (200, (1, 8.20))

    reduceByKey
    reduce
    (100, (3, 66.25)) (200, (1, 8.20)) (300, (2, 58.30)) again
    after
    shuffle

    (300, [42.10, 16.20])

    (100, [22.25, 12.40, 31.60]) (200, [8.20]) (300, [42.10, 16.20])


    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)
    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    ma
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)

    reduceByKe
    (100, [22.25, 12.40, 31.60]) (200, [8.20]) (300, [42.10, 16.20])
    !
    !

    !
    !

    !
    !
    !
    !
    !
    !
    !
    !

    !
    !

    !
    !
    !
    !
    ! !
    ! !
    ! !
    ! !
    ! !
    ! !
    ! !
    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)
    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    ma
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)

    reduceByKe
    (100, [22.25, 12.40, 31.60]) (200, [8.20]) (300, [42.10, 16.20])
    CFFPurchase(100, ”Geneva”, 22.25) CFFPurchase(100, ”Fribourg”, 12.40) CFFPurchase(300, ”Zurich”, 42.10)
    CFFPurchase(100, ”Lucerne”, 31.60) CFFPurchase(200, ”St. Gallen”, 8.20) CFFPurchase(300, ”Basel”, 16.20)

    ma
    (100, 22.25) (100, 12.40) (300, 42.10)
    (100, 31.60) (200, 8.20) (300, 16.20)

    reduceByKe
    (100, [22.25, 12.40, 31.60]) (200, [8.20]) (300, [42.10, 16.20])

    almost a 9x speedup over


    purchasePerMonthSlowLarge!
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !

    !
    !

    You might also like