Class ADAG


  • public class ADAG
    extends java.lang.Object
     This class provides the Java API to create DAX files.
    
     The DAX XML SCHEMA is available at http://pegasus.isi.edu/schema/dax-3.3.xsd
     and documentation available at http://pegasus.isi.edu/wms/docs/schemas/dax-3.3/dax-3.3.html
    
     The DAX consists of 6 parts the first 4 are optional and the last is optional.
     
    1. file:Used as "In DAX" Replica Catalog (Optional)

    2. executable: Used as "In DAX" Transformation Catalog (Optional)

    3. transformation: Used to describe compound executables. i.e. Executable depending on other executables (Optional)

    4. job|dax|dag: Used to describe a single job or sub dax or sub dax. Atleast 1 required.

    5. child: The dependency section to describe dependencies between job|dax|dag elements. (Optional)

     To generate an example DIAMOND DAX run the ADAG Class as shown below
     java ADAG filename
     NOTE: This is an illustrative example only. Please see examples directory for a working example
    
     Shown below are some of the steps in creating a  DIAMOND DAX.
     
    1. Create a new ADAG object

      ADAG dax = new ADAG("test");

    2. Add notifications to the workflow

      j3.addNotification(WHEN.start,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");

      j3.addNotification(WHEN.at_end,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");

    3. Create a File object
      You only need to add entries to this section if you want to use an "IN-DAX" Replica Catalog"

      File fa = new File("f.a");

      1. Add MetaData entry to the file objects

        fa.addMetaData("string", "foo", "bar");
        fa.addMetaData("int", "num", "1");

      2. Add Profile entry to the file objects

        fa.addProfile("env", "FOO", "/usr/bar");
        fa.addProfile("globus", "walltime", "40");

      3. Add PFN to the File object

        fa.addPhysicalFile("file:///scratch/f.a", "local");

      4. Add the File object to the Replica Catalog section of the DAX

        dax.addFile(fa);

    4. Create an Executable object
      You only need to add entries to this section if you want to use an "IN-DAX" Replica Catalog"

      Executable preprocess = new Executable("pegasus", "preproces", "1.0");
      1. Set the Executable.ARCH and Executable.OS for the executable. Default is x86 and LINUX

        preprocess.setArchitecture(Executable.ARCH.x86).setOS(Executable.OS.LINUX);

      2. Set the executable as available to be staged. Default is installed executable

        preprocess.unsetInstalled();

      3. Add the physical location PFN of the executable. In case of stageable executables the path should be a url

        preprocess.addPhysicalFile(new PFN("file:///opt/pegasus/default/bin/keg"));

      4. Add Profile and MetaData objects to the executable

        preprocess.addProfile(Profile.NAMESPACE.globus, "walltime", "120");
        preprocess.addMetaData("string", "project", "pegasus");

      5. Add the Executable object to the ADAG object

        dax.addExecutable(preprocess);

    5. Create a Transformation object : compound Executable (Executable depending on other executable and files)

      Transformation diamond = new Transformation("pegasus", "diamond", "1.0");
      1. Add the sub executable for this transformation

        diamond.uses(preprocess).uses(findrange).uses(analyze);

      2. Add the sub files(e.g config files) for this transformation

        diamond.uses(new File("config", File.LINK.INPUT));

      3. Finally Add the Transformation to the ADAG object

        dax.addTransformation(diamond);

    6. Create a Job object

      Job j1 = new Job("j1", "pegasus", "preprocess", "1.0", "j1");
      1. Add Arguments to the job object

        j1.addArgument("-a","preprocess")
        j1.addArgument("-T","60").addArgument("-i",fa);
        j1.addArgument("-o").addArgument(fb1).addArgument(fb2);

      2. Add the Files that are used by this job

        j1.uses(fa, File.LINK.INPUT);
        j1.uses(fb1, File.LINK.OUTPUT);
        j1.uses(new File("f.b2"), File.LINK.OUTPUT);

      3. Add the Notifications to this job

        j3.addNotification(WHEN.start,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");
        j3.addNotification(WHEN.at_end,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");

      4. Add Profiles to the job

        j1.addProfile(Profile.NAMESPACE.dagman, "pre", "20");

      5. Add the Job object to ADAG

        dax.addJob(j1);

    7. Add a DAG object

      DAG j2 = new DAG("j2", "findrange.dag", "j2");
      j2.uses(new File("f.b1"), File.LINK.INPUT);
      j2.uses(new File("f.c1"), File.LINK.OUTPUT);
      j2.addProfile(Profile.NAMESPACE.dagman, "pre", "20");
      j2.addProfile("condor", "universe", "vanilla");
      dax.addDAG(j2);

    8. Add a DAX job object.

      DAX j3 = new DAX("j3", "findrange.dax", "j3");
      j3.addArgument("--site").addArgument("local");
      j3.uses(new File("f.b2"), File.LINK.INPUT);
      j3.uses(new File("f.c2"), File.LINK.OUTPUT);
      j3.addProfile("ENV", "HAHA", "YADAYADAYADA");
      dax.addDAX(j3);

    9. Add the Job dependencies

      Dependencies can be added by specifiying the job id's like so

      dax.addDependency("j1", "j2", "1-2").addDependency("j1", "j3", "1-3");

      or by specifying the job|dax|dag objects directly as below

      dax.addDependency(j1,j3);

    10. Finally write the dax to a file

      dax.writeToFile("diamond.dax");
    Version:
    $Revision$
    Author:
    Gaurang Mehta gmehta at isi dot edu
    • Field Summary

      Fields 
      Modifier and Type Field Description
      private int mCount
      The Count of the number of dax objects : N
      private java.util.Map<java.lang.String,​java.util.Set<Edge>> mDependencies
      Map of Dependencies between Job,DAX,DAG objects.
      private java.util.Set<Executable> mExecutables
      The list of Executable objects
      private java.util.List<File> mFiles
      The list of edu.isi.pegasus.planner.dax.File objects
      private int mIndex
      The Index of the dax object.
      private java.util.List<Invoke> mInvokes
      List of Notification objects
      private java.util.Map<java.lang.String,​AbstractJob> mJobs
      The List of Job,DAX and DAG objects
      private java.util.List<DAG> mLDAGs  
      private java.util.List<DAX> mLDAXs  
      private java.util.List<Job> mLJobs  
      private LogManager mLogger  
      private java.lang.String mName
      The Name / Label of the DAX
      private java.util.Set<Transformation> mTransformations
      The List of Transformation objects
      private XMLWriter mWriter
      Handle the XML writer
      static java.lang.String SCHEMA_LOCATION
      The "not-so-official" location URL of the DAX schema definition.
      static java.lang.String SCHEMA_NAMESPACE
      The "official" namespace URI of the site catalog schema.
      static java.lang.String SCHEMA_NAMESPACE_XSI
      XSI SCHEMA NAMESPACE
      static java.lang.String SCHEMA_VERSION
      The version to report.
    • Constructor Summary

      Constructors 
      Constructor Description
      ADAG​(java.lang.String name)
      The Simple constructor for the DAX object
      ADAG​(java.lang.String name, int index, int count)
      DAX Constructor
    • Field Detail

      • SCHEMA_NAMESPACE

        public static final java.lang.String SCHEMA_NAMESPACE
        The "official" namespace URI of the site catalog schema.
        See Also:
        Constant Field Values
      • SCHEMA_NAMESPACE_XSI

        public static final java.lang.String SCHEMA_NAMESPACE_XSI
        XSI SCHEMA NAMESPACE
        See Also:
        Constant Field Values
      • SCHEMA_LOCATION

        public static final java.lang.String SCHEMA_LOCATION
        The "not-so-official" location URL of the DAX schema definition.
        See Also:
        Constant Field Values
      • SCHEMA_VERSION

        public static final java.lang.String SCHEMA_VERSION
        The version to report.
        See Also:
        Constant Field Values
      • mName

        private java.lang.String mName
        The Name / Label of the DAX
      • mIndex

        private int mIndex
        The Index of the dax object. I out of N
      • mCount

        private int mCount
        The Count of the number of dax objects : N
      • mLJobs

        private java.util.List<Job> mLJobs
      • mLDAGs

        private java.util.List<DAG> mLDAGs
      • mLDAXs

        private java.util.List<DAX> mLDAXs
      • mExecutables

        private java.util.Set<Executable> mExecutables
        The list of Executable objects
        See Also:
        Executable
      • mFiles

        private java.util.List<File> mFiles
        The list of edu.isi.pegasus.planner.dax.File objects
        See Also:
        File
      • mDependencies

        private java.util.Map<java.lang.String,​java.util.Set<Edge>> mDependencies
        Map of Dependencies between Job,DAX,DAG objects. Map key is a string that holds the child element reference, the value is a List of Parent objects
        See Also:
        Parent
      • mInvokes

        private java.util.List<Invoke> mInvokes
        List of Notification objects
      • mWriter

        private XMLWriter mWriter
        Handle the XML writer
    • Constructor Detail

      • ADAG

        public ADAG​(java.lang.String name)
        The Simple constructor for the DAX object
        Parameters:
        name - DAX LABEL
      • ADAG

        public ADAG​(java.lang.String name,
                    int index,
                    int count)
        DAX Constructor
        Parameters:
        name - DAX Label
        index - Index of DAX out of N DAX's
        count - Number of DAXS in a group
    • Method Detail

      • getName

        public java.lang.String getName()
        Return the name/label of the dax
        Returns:
      • getIndex

        public int getIndex()
      • getCount

        public int getCount()
        Returns the total count of the dax collection. (legacy)
        Returns:
        int
      • addInvoke

        public ADAG addInvoke​(Invoke.WHEN when,
                              java.lang.String what)
        Add a Notification for this Workflow
        Parameters:
        when -
        what -
        Returns:
        ADAG
      • addNotification

        public ADAG addNotification​(Invoke.WHEN when,
                                    java.lang.String what)
        Add a Notification for this Workflow
        Parameters:
        when -
        what -
        Returns:
        ADAG
      • addInvoke

        public ADAG addInvoke​(Invoke invoke)
        Add a Notification for this Workflow
        Parameters:
        invoke -
        Returns:
        ADAG
      • addNotification

        public ADAG addNotification​(Invoke invoke)
        Add a Notification for this Workflow
        Parameters:
        invoke -
        Returns:
        ADAG
      • addInvokes

        public ADAG addInvokes​(java.util.List<Invoke> invokes)
        Add a List of Notifications for this Workflow
        Parameters:
        invokes -
        Returns:
        ADAG
      • addNotifications

        public ADAG addNotifications​(java.util.List<Invoke> invokes)
        Add a List of Notifications for this Workflow
        Parameters:
        invokes -
        Returns:
        ADAG
      • getInvoke

        public java.util.List<Invoke> getInvoke()
        Returns a list of Invoke objects associated with the workflow
        Returns:
      • getNotification

        public java.util.List<Invoke> getNotification()
        Returns a list of Invoke objects associated with the workflow. Same as getInvoke()
        Returns:
      • addFile

        public ADAG addFile​(File file)
        Add a RC File object to the top of the DAX.
        Parameters:
        file - File object to be added to the RC section
        Returns:
        ADAG
        See Also:
        File
      • addFiles

        public ADAG addFiles​(java.util.List<File> files)
        Add Files to the RC Section on top of the DAX
        Parameters:
        files - List List of file objects to be added to the RC Section
        Returns:
        ADAG
        See Also:
        File
      • getFiles

        public java.util.List<File> getFiles()
        Returns a list of File objects defined as the inDax Replica Catalog
        Returns:
      • addExecutable

        public ADAG addExecutable​(Executable executable)
        Add Executable to the DAX
        Parameters:
        executable - Executable to be added
        Returns:
        ADAG
        See Also:
        Executable
      • addExecutables

        public ADAG addExecutables​(java.util.List<Executable> executables)
        Add Multiple Executable objects to the DAX
        Parameters:
        executables - List of Executable objects to be added
        Returns:
        ADAG
        See Also:
        Executable
      • getExecutables

        public java.util.Set<Executable> getExecutables()
        Returns a set of Executable Objects stored as part of the inDAX Transformation Catalog;
        Returns:
      • containsExecutable

        public boolean containsExecutable​(Executable executable)
        Checks if a given executable exists in the DAX based Transformation Catalog
        Parameters:
        executable -
        Returns:
        boolean
      • addTransformation

        public ADAG addTransformation​(Transformation transformation)
        Add Transformation to the DAX
        Parameters:
        transformation - Transformation object to be added
        Returns:
        ADAG
        See Also:
        Transformation
      • addTransformations

        public ADAG addTransformations​(java.util.List<Transformation> transformations)
        Add Multiple Transformation to the DAX
        Parameters:
        transformations - List of Transformation objects
        Returns:
        ADAG
        See Also:
        Transformation
      • containsTransformation

        public boolean containsTransformation​(Transformation transformation)
        Checks if a given Transformation exists in the DAX based Transformation Catalog
        Parameters:
        transformation - Transformation
        Returns:
        boolean
      • getTransformations

        public java.util.Set<Transformation> getTransformations()
        Returns a set of Transformation Objects (complex executables) stored in the DAX based Transformation Catalog
        Returns:
      • addAbstractJob

        private ADAG addAbstractJob​(AbstractJob ajob)
        Add AbstractJob to the DAX
        Parameters:
        ajob - AbstractJob
        Returns:
        ADAG
        See Also:
        Job, DAG, DAX, AbstractJob
      • addAbstractJobs

        private ADAG addAbstractJobs​(java.util.List<AbstractJob> ajobs)
        Add AbstractJobs to the DAX
        Parameters:
        ajobs - AbstractJob
        Returns:
        ADAG
        See Also:
        Job, DAG, DAX, AbstractJob
      • getAbstractJob

        private AbstractJob getAbstractJob​(java.lang.String ajobid)
        Returns an abstract Job with id ajobid if present otherwise null.
        Parameters:
        ajobid -
        Returns:
      • containsAbstractJob

        private boolean containsAbstractJob​(AbstractJob ajob)
        Check if an abstractjob exists in the DAX
        Parameters:
        ajob -
        Returns:
      • containsAbstractJobId

        private boolean containsAbstractJobId​(java.lang.String ajobid)
        Check if a jobid exists in the DAX
        Parameters:
        ajobid -
        Returns:
      • addJob

        public ADAG addJob​(Job job)
        Add Job to the DAX
        Parameters:
        job -
        Returns:
        ADAG
        See Also:
        Job, AbstractJob
      • addJobs

        public ADAG addJobs​(java.util.List<Job> jobs)
        Add multiple Jobs to the DAX
        Parameters:
        jobs -
        Returns:
        ADAG
        See Also:
        Job, AbstractJob
      • containsJob

        public boolean containsJob​(Job job)
        Check if a job exists in the DAX
        Parameters:
        job -
        Returns:
      • containsJobId

        public boolean containsJobId​(java.lang.String jobid)
        Check if a jobid exists in the DAX
        Parameters:
        jobid -
        Returns:
      • getJob

        public Job getJob​(java.lang.String jobid)
        Returns a Job object with id jobid if present otherwise null.
        Parameters:
        jobid -
        Returns:
      • getJobs

        public java.util.List<Job> getJobs()
        Get a list of all the DAG jobs.
        Returns:
      • getDAXs

        public java.util.List<DAX> getDAXs()
        Get a list of all the DAX jobs.
        Returns:
      • getDAX

        public DAX getDAX​(java.lang.String daxid)
        Returns a DAX object with id daxid if present otherwise null.
        Parameters:
        daxid -
        Returns:
      • getDAGs

        public java.util.List<DAG> getDAGs()
        Get a list of all the DAG jobs.
        Returns:
      • getDAG

        public DAG getDAG​(java.lang.String dagid)
        Returns a DAG object with id dagid if present otherwise null.
        Parameters:
        dagid -
        Returns:
      • addDAG

        public ADAG addDAG​(DAG dag)
        Add a DAG job to the DAX
        Parameters:
        dag - the DAG to be added
        Returns:
        ADAG
        See Also:
        DAG, AbstractJob
      • addDAGs

        public ADAG addDAGs​(java.util.List<DAG> dags)
        Add multiple DAG jobs to the DAX
        Parameters:
        dags - List of DAG jobs to be added
        Returns:
        ADAG
        See Also:
        DAG, AbstractJob
      • containsDAG

        public boolean containsDAG​(DAG dag)
        Check if a DAG job exists in the DAX
        Parameters:
        dag -
        Returns:
      • containsDAGId

        public boolean containsDAGId​(java.lang.String dagid)
        Check if a DAG job id exists in the DAX
        Parameters:
        dagid -
        Returns:
      • addDAX

        public ADAG addDAX​(DAX dax)
        Add a DAX job to the DAX
        Parameters:
        dax - DAX to be added
        Returns:
        ADAG
        See Also:
        DAX, AbstractJob
      • addDAXs

        public ADAG addDAXs​(java.util.List<DAX> daxs)
        Add multiple DAX jobs to the DAX
        Parameters:
        daxs - LIST of DAX jobs to be added
        Returns:
        ADAG
        See Also:
        DAX, AbstractJob
      • containsDAX

        public boolean containsDAX​(DAX dax)
        Check if a DAX job exists in the DAX
        Parameters:
        dax -
        Returns:
      • containsDAXId

        public boolean containsDAXId​(java.lang.String daxid)
        Check if a DAX job id exists in the DAX
        Parameters:
        daxid -
        Returns:
      • addDependency

        public ADAG addDependency​(java.lang.String parent,
                                  java.lang.String child)
        Add a parent child dependency between two jobs,dax,dag
        Parameters:
        parent - String job,dax,dag id
        child - String job,dax,dag,id
        Returns:
        ADAG
      • addDependency

        public ADAG addDependency​(AbstractJob parent,
                                  AbstractJob child)
        Add a parent child dependency between two jobs,dax,dag
        Parameters:
        parent - Job|DAX|DAG object
        child - Job|DAX|DAG object
        Returns:
      • addDependency

        public ADAG addDependency​(java.lang.String parent,
                                  java.lang.String child,
                                  java.lang.String label)
        Add a parent child dependency with a dependency label
        Parameters:
        parent - String job,dax,dag id
        child - String job,dax,dag id
        label - String dependency label
        Returns:
        ADAG
      • getEdges

        public java.util.Set<Edge> getEdges​(java.lang.String child)
        Returns a list of Edge objects for a child job/dax/dag id. Returns an empty set if the child does not have any parents Returns null if the child is not a valid job/dax/dag id
        Parameters:
        child -
        Returns:
      • getEdges

        public java.util.Set<Edge> getEdges()
        Returns a Set of all the Edge objects for the DAX. Returns empty if no dependencies.
        Parameters:
        child -
        Returns:
      • addDependency

        public ADAG addDependency​(AbstractJob parent,
                                  AbstractJob child,
                                  java.lang.String label)
        Add a parent child dependency with a dependency label
        Parameters:
        parent - Job|DAX|DAG object
        child - Job|DAX|DAG object
        label - String label for annotation
        Returns:
        ADAG
      • writeToFile

        public void writeToFile​(java.lang.String daxfile)
        Generate a DAX File out of this object;
        Parameters:
        daxfile - The file to write the DAX to
      • writeToSTDOUT

        public void writeToSTDOUT()
        Generate a DAX representation on STDOUT.
      • writeToWriter

        public void writeToWriter​(java.io.Writer writer,
                                  boolean close)
        Generate a DAX representation and pipe it into the Writer
        Parameters:
        writer - A Writer object
        close - Whether writer should be closed on return.
      • toXML

        public void toXML​(XMLWriter writer)
        Generates a DAX representation.
        Parameters:
        writer -
      • main

        public static void main​(java.lang.String[] args)
        Create an example DIAMOND DAX
        Parameters:
        args -
      • Diamond

        private static ADAG Diamond()