Class Horizontal


  • public class Horizontal
    extends BFS
    Horizontal based partitioning scheme, that allows the user to configure the number of partitions per transformation name per level. To set the size of the partition per transformation, the following properties need to be set
           pegasus.partitioner.horizontal.collapse.[txName]
           pegasus.partitioner.horizontal.bundle.[txName]
     
    The bundle value designates the number of partitions per transformation per level. The collapse values designates the number of nodes in a partitioning referring to a particular transformation. If both are specified, then bundle value takes precedence.
    Version:
    $Revision$
    Author:
    Karan Vahi
    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      private static class  Horizontal.GraphNodeComparator
      A GraphNode comparator, that allows us to compare nodes according to the transformation logical names.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected void constructLevelRelations​(Callback c, int parent, int child)
      Calls out to the callback with appropriate relations between the partitions constructed for the levels.
      protected void constructPartitions​(Callback c, java.util.List nodes, int level)
      Given a list of jobs, constructs (one or more) partitions out of it.
      protected void constructPartitions​(Callback c, java.util.List nodes, int level, java.lang.String name)
      Given a list of jobs, constructs (one or more) partitions out of it.
      protected Partition createPartition​(java.util.List nodes)
      Creates a partition out of a list of nodes.
      java.lang.String description()
      Returns a textual description of the partitioner implementation.
      protected void done​(Callback c)
      Indicates that we are done with the traversal of the graph.
      protected int[] getCollapseFactor​(java.lang.String txName, int size)
      Returns the collapse factor, that is used to determine the number of nodes going in a partition.
      private java.lang.String getPartitionID​(int id)
      Constructs the id for the partition.
      private int idCounter()
      Returns the current value of the ID counter.
      private void incrementIDCounter()
      Increments the ID counter by 1.
      private java.util.Comparator nodeComparator()
      Singleton access to the job comparator.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • DESCRIPTION

        public static final java.lang.String DESCRIPTION
        A short description about the partitioner.
        See Also:
        Constant Field Values
      • DEFAULT_COLLAPSE_FACTOR

        public static final int DEFAULT_COLLAPSE_FACTOR
        The default collapse factor for collapsing jobs with same logical name scheduled onto the same execution pool.
        See Also:
        Constant Field Values
      • mPartitionMap

        private java.util.Map mPartitionMap
        A map indexed by the partition ID. Each value is a partition object.
      • mIDCounter

        private int mIDCounter
        The global counter that is used to assign ID's to the partitions.
    • Constructor Detail

      • Horizontal

        public Horizontal​(GraphNode root,
                          java.util.Map graph,
                          PegasusProperties properties)
        The overloaded constructor.
        Parameters:
        root - the dummy root node of the graph.
        graph - the map containing all the nodes of the graph keyed by the logical id of the nodes.
        properties - the properties passed to the planner.
    • Method Detail

      • nodeComparator

        private java.util.Comparator nodeComparator()
        Singleton access to the job comparator.
        Returns:
        the job comparator.
      • description

        public java.lang.String description()
        Returns a textual description of the partitioner implementation.
        Overrides:
        description in class BFS
        Returns:
        a short textual description
      • constructPartitions

        protected void constructPartitions​(Callback c,
                                           java.util.List nodes,
                                           int level)
        Given a list of jobs, constructs (one or more) partitions out of it. Calls out to the partitioner callback, for each of the partitions constructed.
        Overrides:
        constructPartitions in class BFS
        Parameters:
        c - the parititoner callback
        nodes - the list of GraphNode objects on a particular level.
        level - the level as determined from the root of the workflow.
      • constructPartitions

        protected void constructPartitions​(Callback c,
                                           java.util.List nodes,
                                           int level,
                                           java.lang.String name)
        Given a list of jobs, constructs (one or more) partitions out of it. Calls out to the partitioner callback, for each of the partitions constructed.
        Parameters:
        c - the parititoner callback
        nodes - the list of GraphNode objects on a particular level, referring to the same transformation underneath.
        level - the level as determined from the root of the workflow.
        name - the transformation name
      • constructLevelRelations

        protected void constructLevelRelations​(Callback c,
                                               int parent,
                                               int child)
        Calls out to the callback with appropriate relations between the partitions constructed for the levels. This is an empty implementation, as we do our own book-keeping in this partitioner to determine the relations between the partitions.
        Overrides:
        constructLevelRelations in class BFS
        Parameters:
        c - the parititoner callback
        parent - the parent level
        child - the child level.
        See Also:
        done( Callback )
      • done

        protected void done​(Callback c)
        Indicates that we are done with the traversal of the graph. Determines the relations between the partitions constructed and calls out to the appropriate callback function
        Overrides:
        done in class BFS
        Parameters:
        c - the partitioner callback
      • getCollapseFactor

        protected int[] getCollapseFactor​(java.lang.String txName,
                                          int size)
        Returns the collapse factor, that is used to determine the number of nodes going in a partition. The collapse factor is determined by getting the collapse and the bundle values specified for the transformations in the properties file. There are two orthogonal notions of bundling and collapsing. In case the bundle key is specified, it ends up overriding the collapse key, and the bundle value is used to generate the collapse values. If both are not specified or null, then collapseFactor is set to size.
        Parameters:
        txName - the logical transformation name
        size - the number of jobs that refer to the same logical transformation and are scheduled on the same execution pool.
        Returns:
        int array of size 2 where :- int[0] is the the collapse factor (number of nodes in a partition) int[1] is the number of parititons for whom collapsing is int[0] + 1.
      • createPartition

        protected Partition createPartition​(java.util.List nodes)
        Creates a partition out of a list of nodes. Also stores it in the internal partition map to track partitions later on. Associates the partition ID with each of the nodes making the partition also.
        Parameters:
        nodes - the list of GraphNodes making the partition.
        Returns:
        the partition out of those nodes.
      • incrementIDCounter

        private void incrementIDCounter()
        Increments the ID counter by 1.
      • idCounter

        private int idCounter()
        Returns the current value of the ID counter.
      • getPartitionID

        private java.lang.String getPartitionID​(int id)
        Constructs the id for the partition.
        Parameters:
        id - an integer ID.
        Returns:
        the ID for the Partition.