Scribd Logo
Upload

Welcome to Scribd!

  • Spark Job Scheduling: Figure 2-2. A Simple Diagram of Dependencies Between Partitions For Narrow Transformations

    Uploaded by

    OccasionalVisitor
    6 views1 page
    The document discusses diagrams showing dependencies between partitions for narrow and wide transformations in Spark. Figure 2-2 shows narrow dependencies where child partitions depend on a single parent partition. Figure 2-3 shows wide dependencies where child partitions depend on an arbitrary set of parent partitions. The dependencies for operations like groupByKey and sortByKey follow the wide pattern. Join functions can have either wide or narrow dependencies depending on how the two parent RDDs are partitioned.

    Original Description:

    Original Title

    spark_dependenciesk

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document discusses diagrams showing dependencies between partitions for narrow and wide transformations in Spark. Figure 2-2 shows narrow dependencies where child partitions depend on a single parent partition. Figure 2-3 shows wide dependencies where child partitions depend on an arbitrary set of parent partitions. The dependencies for operations like groupByKey and sortByKey follow the wide pattern. Join functions can have either wide or narrow dependencies depending on how the two parent RDDs are partitioned.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    6 views1 page

    Spark Job Scheduling: Figure 2-2. A Simple Diagram of Dependencies Between Partitions For Narrow Transformations

    Uploaded by

    OccasionalVisitor
    The document discusses diagrams showing dependencies between partitions for narrow and wide transformations in Spark. Figure 2-2 shows narrow dependencies where child partitions depend on a single parent partition. Figure 2-3 shows wide dependencies where child partitions depend on an arbitrary set of parent partitions. The dependencies for operations like groupByKey and sortByKey follow the wide pattern. Join functions can have either wide or narrow dependencies depending on how the two parent RDDs are partitioned.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 1

    Figure 2-2.

    A simple diagram of dependencies between partitions for narrow


    transformations

    Figure 2-3 shows wide dependencies between partitions. In this case the child parti‐
    tions (shown at the bottom of Figure 2-3) depend on an arbitrary set of parent parti‐
    tions. The wide dependencies (displayed as red arrows) cannot be known fully before
    the data is evaluated. In contrast to the coalesce operation, data is partitioned
    according to its value. The dependency graph for any operations that cause a shuffle
    (such as groupByKey, reduceByKey, sort, and sortByKey) follows this pattern.

    Figure 2-3. A simple diagram of dependencies between partitions for wide


    transformations

    The join functions are a bit more complicated, since they can have wide or narrow
    dependencies depending on how the two parent RDDs are partitioned. We illustrate
    the dependencies in different scenarios for the join operation in “Core Spark Joins”
    on page 73.

    Spark Job Scheduling


    A Spark application consists of a driver process, which is where the high-level Spark
    logic is written, and a series of executor processes that can be scattered across the
    nodes of a cluster. The Spark program itself runs in the driver node and sends some
    instructions to the executors. One Spark cluster can run several Spark applications
    concurrently. The applications are scheduled by the cluster manager and correspond

    Spark Job Scheduling | 19

    You might also like