Class Chain

  • All Implemented Interfaces:
    Refiner, Refiner

    public class Chain
    extends Basic
    This transfer refiner builds upon the Default Refiner. The defaul Refiner allows the transfer of multiple files in a single condor job. However, it adds the stage in transfer nodes in parallel leading to multiple invocation of the globus-url-copy at remote execution pools, while running huge workflows. This refiner, tries to circumvent this problem by chaining up the stagein jobs instead of scheduling in parallel. This works best only when the top level of the workflow requires stage in jobs. The correct way is that the traversal needs to be done breath first in the TransferEngine.java.
    Version:
    $Revision$
    Author:
    Karan Vahi, Gaurang Mehta
    • Field Detail

      • DEFAULT_BUNDLE_FACTOR

        public static final java.lang.String DEFAULT_BUNDLE_FACTOR
        The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for the workflow.
        See Also:
        Constant Field Values
      • mSiteStore

        protected SiteStore mSiteStore
        The handle to the Site Catalog. It is instantiated in this class.
      • mSIBundleMap

        private java.util.Map mSIBundleMap
        The map containing the stage in bundle values indexed by the name of the site. If the bundle value is not specified, then null is stored.
      • mStageInMap

        private java.util.Map mStageInMap
        A map indexed by execution sites. Each value is a SiteTransfer object, that contains the Bundles of stagin transfer jobs.
        See Also:
        Chain.TransferChain
      • DESCRIPTION

        public static final java.lang.String DESCRIPTION
        A short description of the transfer refinement.
        See Also:
        Constant Field Values
    • Constructor Detail

      • Chain

        public Chain​(ADag dag,
                     PegasusBag bag)
        The overloaded constructor.
        Parameters:
        dag - the workflow to which transfer nodes need to be added.
        bag - the bag of initialization objects
    • Method Detail

      • addRelation

        public void addRelation​(java.lang.String parent,
                                java.lang.String child,
                                java.lang.String site,
                                boolean parentNew)
        Adds a new relation to the workflow. In the case when the parent is a transfer job that is added, the parentNew should be set only the first time a relation is added. For subsequent compute jobs that maybe dependant on this, it needs to be set to false.
        Specified by:
        addRelation in interface Refiner
        Overrides:
        addRelation in class Basic
        Parameters:
        parent - the jobname of the parent node of the edge.
        child - the jobname of the child node of the edge.
        site - the execution site where the transfer node is to be run.
        parentNew - the parent node being added, is the new transfer job and is being called for the first time.
      • getSiteBundleValue

        public int getSiteBundleValue​(java.lang.String site,
                                      java.lang.String key)
        Determines the bundle factor for a particular site on the basis of the key associcated with the underlying transfer transformation in the transformation catalog. If none specified in transformation catalog then one is picked up from the site catalog. If the key is not found in the site catalog too , then the global default is returned.
        Parameters:
        site - the site at which the transfer job is being run.
        key - the bundle key whose value needs to be searched.
        Returns:
        the bundle factor.
        See Also:
        DEFAULT_BUNDLE_FACTOR
      • done

        public void done()
        Prints out the bundles and chains that have been constructed.
        Specified by:
        done in interface Refiner
        Overrides:
        done in class Basic
      • getDescription

        public java.lang.String getDescription()
        Returns a textual description of the transfer mode.
        Specified by:
        getDescription in interface Refiner
        Overrides:
        getDescription in class Basic
        Returns:
        a short textual description