Package edu.isi.pegasus.planner.dax
Class ADAG
- java.lang.Object
-
- edu.isi.pegasus.planner.dax.ADAG
-
public class ADAG extends java.lang.Object
This class provides the Java API to create DAX files. The DAX XML SCHEMA is available at http://pegasus.isi.edu/schema/dax-3.3.xsd and documentation available at http://pegasus.isi.edu/wms/docs/schemas/dax-3.3/dax-3.3.html The DAX consists of 6 parts the first 4 are optional and the last is optional.
- file:Used as "In DAX" Replica Catalog (Optional)
- executable: Used as "In DAX" Transformation Catalog (Optional)
- transformation: Used to describe compound executables. i.e. Executable depending on other executables (Optional)
- job|dax|dag: Used to describe a single job or sub dax or sub dax. Atleast 1 required.
- child: The dependency section to describe dependencies between job|dax|dag elements. (Optional)
To generate an example DIAMOND DAX run the ADAG Class as shown below java ADAG filename NOTE: This is an illustrative example only. Please see examples directory for a working example Shown below are some of the steps in creating a DIAMOND DAX.
- Create a new
ADAG
object
ADAG dax = new ADAG("test"); - Add notifications to the
workflow
j3.addNotification(WHEN.start,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");
j3.addNotification(WHEN.at_end,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com"); - Create
a
File
object
You only need to add entries to this section if you want to use an "IN-DAX" Replica Catalog"
File fa = new File("f.a"); - Add
MetaData
entry to the file objects
fa.addMetaData("string", "foo", "bar");
fa.addMetaData("int", "num", "1");
- Add
Profile
entry to the file objects
fa.addProfile("env", "FOO", "/usr/bar");
fa.addProfile("globus", "walltime", "40"); - Add
PFN
to the File object
fa.addPhysicalFile("file:///scratch/f.a", "local"); - Add the File object to the Replica Catalog
section of the DAX
dax.addFile(fa); - Create an
Executable
object
You only need to add entries to this section if you want to use an "IN-DAX" Replica Catalog"
Executable preprocess = new Executable("pegasus", "preproces", "1.0");
- Set the
Executable.ARCH
andExecutable.OS
for the executable. Default is x86 and LINUX
preprocess.setArchitecture(Executable.ARCH.x86).setOS(Executable.OS.LINUX); - Set the executable as available to be staged. Default is
installed executable
preprocess.unsetInstalled(); - Add the physical location
PFN
of the executable. In case of stageable executables the path should be a url
preprocess.addPhysicalFile(new PFN("file:///opt/pegasus/default/bin/keg")); - Add
Profile
andMetaData
objects to the executable
preprocess.addProfile(Profile.NAMESPACE.globus, "walltime", "120");
preprocess.addMetaData("string", "project", "pegasus"); - Add the
Executable
object to theADAG
object
dax.addExecutable(preprocess);
- Set the
- Create a
Transformation
object : compound Executable (Executable depending on other executable and files)
Transformation diamond = new Transformation("pegasus", "diamond", "1.0");
- Add the sub executable for this
transformation
diamond.uses(preprocess).uses(findrange).uses(analyze);
- Add the sub files(e.g config files) for this
transformation
diamond.uses(new File("config", File.LINK.INPUT));
- Finally Add the Transformation to
the
ADAG
object
dax.addTransformation(diamond);
- Add the sub executable for this
transformation
- Create a
Job
object
Job j1 = new Job("j1", "pegasus", "preprocess", "1.0", "j1");
- Add Arguments to the job object
j1.addArgument("-a","preprocess")
j1.addArgument("-T","60").addArgument("-i",fa);
j1.addArgument("-o").addArgument(fb1).addArgument(fb2); - Add the Files that are used by this job
j1.uses(fa, File.LINK.INPUT);
j1.uses(fb1, File.LINK.OUTPUT);
j1.uses(new File("f.b2"), File.LINK.OUTPUT); - Add the Notifications
to this job
j3.addNotification(WHEN.start,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");
j3.addNotification(WHEN.at_end,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com"); - Add
Profile
s to the job
j1.addProfile(Profile.NAMESPACE.dagman, "pre", "20"); - Add the Job object to
ADAG
dax.addJob(j1);
- Add Arguments to the job object
- Add a
DAG
object
DAG j2 = new DAG("j2", "findrange.dag", "j2");
j2.uses(new File("f.b1"), File.LINK.INPUT);
j2.uses(new File("f.c1"), File.LINK.OUTPUT);
j2.addProfile(Profile.NAMESPACE.dagman, "pre", "20");
j2.addProfile("condor", "universe", "vanilla");
dax.addDAG(j2); - Add a
DAX
job object.
DAX j3 = new DAX("j3", "findrange.dax", "j3");
j3.addArgument("--site").addArgument("local");
j3.uses(new File("f.b2"), File.LINK.INPUT);
j3.uses(new File("f.c2"), File.LINK.OUTPUT);
j3.addProfile("ENV", "HAHA", "YADAYADAYADA");
dax.addDAX(j3); - Add the Job dependencies
Dependencies can be added by specifiying the job id's like so
dax.addDependency("j1", "j2", "1-2").addDependency("j1", "j3", "1-3");
or by specifying the job|dax|dag objects directly as below
dax.addDependency(j1,j3); - Finally write the dax to a
file
dax.writeToFile("diamond.dax");
- Version:
- $Revision$
- Author:
- Gaurang Mehta gmehta at isi dot edu
-
-
Field Summary
Fields Modifier and Type Field Description private int
mCount
The Count of the number of dax objects : Nprivate java.util.Map<java.lang.String,java.util.Set<Edge>>
mDependencies
Map of Dependencies between Job,DAX,DAG objects.private java.util.Set<Executable>
mExecutables
The list of Executable objectsprivate java.util.List<File>
mFiles
The list of edu.isi.pegasus.planner.dax.File objectsprivate int
mIndex
The Index of the dax object.private java.util.List<Invoke>
mInvokes
List of Notification objectsprivate java.util.Map<java.lang.String,AbstractJob>
mJobs
The List of Job,DAX and DAG objectsprivate java.util.List<DAG>
mLDAGs
private java.util.List<DAX>
mLDAXs
private java.util.List<Job>
mLJobs
private LogManager
mLogger
private java.lang.String
mName
The Name / Label of the DAXprivate java.util.Set<Transformation>
mTransformations
The List of Transformation objectsprivate XMLWriter
mWriter
Handle the XML writerstatic java.lang.String
SCHEMA_LOCATION
The "not-so-official" location URL of the DAX schema definition.static java.lang.String
SCHEMA_NAMESPACE
The "official" namespace URI of the site catalog schema.static java.lang.String
SCHEMA_NAMESPACE_XSI
XSI SCHEMA NAMESPACEstatic java.lang.String
SCHEMA_VERSION
The version to report.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description private ADAG
addAbstractJob(AbstractJob ajob)
Add AbstractJob to the DAXprivate ADAG
addAbstractJobs(java.util.List<AbstractJob> ajobs)
Add AbstractJobs to the DAXADAG
addDAG(DAG dag)
Add a DAG job to the DAXADAG
addDAGs(java.util.List<DAG> dags)
Add multiple DAG jobs to the DAXADAG
addDAX(DAX dax)
Add a DAX job to the DAXADAG
addDAXs(java.util.List<DAX> daxs)
Add multiple DAX jobs to the DAXADAG
addDependency(AbstractJob parent, AbstractJob child)
Add a parent child dependency between two jobs,dax,dagADAG
addDependency(AbstractJob parent, AbstractJob child, java.lang.String label)
Add a parent child dependency with a dependency labelADAG
addDependency(java.lang.String parent, java.lang.String child)
Add a parent child dependency between two jobs,dax,dagADAG
addDependency(java.lang.String parent, java.lang.String child, java.lang.String label)
Add a parent child dependency with a dependency labelADAG
addExecutable(Executable executable)
Add Executable to the DAXADAG
addExecutables(java.util.List<Executable> executables)
Add Multiple Executable objects to the DAXADAG
addFile(File file)
Add a RC File object to the top of the DAX.ADAG
addFiles(java.util.List<File> files)
Add Files to the RC Section on top of the DAXADAG
addInvoke(Invoke invoke)
Add a Notification for this WorkflowADAG
addInvoke(Invoke.WHEN when, java.lang.String what)
Add a Notification for this WorkflowADAG
addInvokes(java.util.List<Invoke> invokes)
Add a List of Notifications for this WorkflowADAG
addJob(Job job)
Add Job to the DAXADAG
addJobs(java.util.List<Job> jobs)
Add multiple Jobs to the DAXADAG
addNotification(Invoke invoke)
Add a Notification for this WorkflowADAG
addNotification(Invoke.WHEN when, java.lang.String what)
Add a Notification for this WorkflowADAG
addNotifications(java.util.List<Invoke> invokes)
Add a List of Notifications for this WorkflowADAG
addTransformation(Transformation transformation)
Add Transformation to the DAXADAG
addTransformations(java.util.List<Transformation> transformations)
Add Multiple Transformation to the DAXprivate boolean
containsAbstractJob(AbstractJob ajob)
Check if an abstractjob exists in the DAXprivate boolean
containsAbstractJobId(java.lang.String ajobid)
Check if a jobid exists in the DAXboolean
containsDAG(DAG dag)
Check if a DAG job exists in the DAXboolean
containsDAGId(java.lang.String dagid)
Check if a DAG job id exists in the DAXboolean
containsDAX(DAX dax)
Check if a DAX job exists in the DAXboolean
containsDAXId(java.lang.String daxid)
Check if a DAX job id exists in the DAXboolean
containsExecutable(Executable executable)
Checks if a given executable exists in the DAX based Transformation Catalogboolean
containsJob(Job job)
Check if a job exists in the DAXboolean
containsJobId(java.lang.String jobid)
Check if a jobid exists in the DAXboolean
containsTransformation(Transformation transformation)
Checks if a given Transformation exists in the DAX based Transformation Catalogprivate static ADAG
Diamond()
private AbstractJob
getAbstractJob(java.lang.String ajobid)
Returns an abstract Job with id ajobid if present otherwise null.int
getCount()
Returns the total count of the dax collection.DAG
getDAG(java.lang.String dagid)
Returns a DAG object with id dagid if present otherwise null.java.util.List<DAG>
getDAGs()
Get a list of all the DAG jobs.DAX
getDAX(java.lang.String daxid)
Returns a DAX object with id daxid if present otherwise null.java.util.List<DAX>
getDAXs()
Get a list of all the DAX jobs.java.util.Set<Edge>
getEdges()
Returns a Set of all the Edge objects for the DAX.java.util.Set<Edge>
getEdges(java.lang.String child)
Returns a list of Edge objects for a child job/dax/dag id.java.util.Set<Executable>
getExecutables()
Returns a set of Executable Objects stored as part of the inDAX Transformation Catalog;java.util.List<File>
getFiles()
Returns a list of File objects defined as the inDax Replica Catalogint
getIndex()
java.util.List<Invoke>
getInvoke()
Returns a list of Invoke objects associated with the workflowJob
getJob(java.lang.String jobid)
Returns a Job object with id jobid if present otherwise null.java.util.List<Job>
getJobs()
Get a list of all the DAG jobs.java.lang.String
getName()
Return the name/label of the daxjava.util.List<Invoke>
getNotification()
Returns a list of Invoke objects associated with the workflow.java.util.Set<Transformation>
getTransformations()
Returns a set of Transformation Objects (complex executables) stored in the DAX based Transformation Catalogstatic void
main(java.lang.String[] args)
Create an example DIAMOND DAXvoid
toXML(XMLWriter writer)
Generates a DAX representation.void
writeToFile(java.lang.String daxfile)
Generate a DAX File out of this object;void
writeToSTDOUT()
Generate a DAX representation on STDOUT.void
writeToWriter(java.io.Writer writer, boolean close)
Generate a DAX representation and pipe it into the Writer
-
-
-
Field Detail
-
SCHEMA_NAMESPACE
public static final java.lang.String SCHEMA_NAMESPACE
The "official" namespace URI of the site catalog schema.- See Also:
- Constant Field Values
-
SCHEMA_NAMESPACE_XSI
public static final java.lang.String SCHEMA_NAMESPACE_XSI
XSI SCHEMA NAMESPACE- See Also:
- Constant Field Values
-
SCHEMA_LOCATION
public static final java.lang.String SCHEMA_LOCATION
The "not-so-official" location URL of the DAX schema definition.- See Also:
- Constant Field Values
-
SCHEMA_VERSION
public static final java.lang.String SCHEMA_VERSION
The version to report.- See Also:
- Constant Field Values
-
mName
private java.lang.String mName
The Name / Label of the DAX
-
mIndex
private int mIndex
The Index of the dax object. I out of N
-
mCount
private int mCount
The Count of the number of dax objects : N
-
mJobs
private java.util.Map<java.lang.String,AbstractJob> mJobs
The List of Job,DAX and DAG objects- See Also:
DAG
,DAX
,Job
,AbstractJob
-
mLJobs
private java.util.List<Job> mLJobs
-
mLDAGs
private java.util.List<DAG> mLDAGs
-
mLDAXs
private java.util.List<DAX> mLDAXs
-
mTransformations
private java.util.Set<Transformation> mTransformations
The List of Transformation objects- See Also:
Transformation
-
mExecutables
private java.util.Set<Executable> mExecutables
The list of Executable objects- See Also:
Executable
-
mFiles
private java.util.List<File> mFiles
The list of edu.isi.pegasus.planner.dax.File objects- See Also:
File
-
mDependencies
private java.util.Map<java.lang.String,java.util.Set<Edge>> mDependencies
Map of Dependencies between Job,DAX,DAG objects. Map key is a string that holds the child element reference, the value is a List of Parent objects- See Also:
Parent
-
mInvokes
private java.util.List<Invoke> mInvokes
List of Notification objects
-
mWriter
private XMLWriter mWriter
Handle the XML writer
-
mLogger
private LogManager mLogger
-
-
Constructor Detail
-
ADAG
public ADAG(java.lang.String name)
The Simple constructor for the DAX object- Parameters:
name
- DAX LABEL
-
ADAG
public ADAG(java.lang.String name, int index, int count)
DAX Constructor- Parameters:
name
- DAX Labelindex
- Index of DAX out of N DAX'scount
- Number of DAXS in a group
-
-
Method Detail
-
getName
public java.lang.String getName()
Return the name/label of the dax- Returns:
-
getIndex
public int getIndex()
-
getCount
public int getCount()
Returns the total count of the dax collection. (legacy)- Returns:
- int
-
addInvoke
public ADAG addInvoke(Invoke.WHEN when, java.lang.String what)
Add a Notification for this Workflow- Parameters:
when
-what
-- Returns:
- ADAG
-
addNotification
public ADAG addNotification(Invoke.WHEN when, java.lang.String what)
Add a Notification for this Workflow- Parameters:
when
-what
-- Returns:
- ADAG
-
addInvoke
public ADAG addInvoke(Invoke invoke)
Add a Notification for this Workflow- Parameters:
invoke
-- Returns:
- ADAG
-
addNotification
public ADAG addNotification(Invoke invoke)
Add a Notification for this Workflow- Parameters:
invoke
-- Returns:
- ADAG
-
addInvokes
public ADAG addInvokes(java.util.List<Invoke> invokes)
Add a List of Notifications for this Workflow- Parameters:
invokes
-- Returns:
- ADAG
-
addNotifications
public ADAG addNotifications(java.util.List<Invoke> invokes)
Add a List of Notifications for this Workflow- Parameters:
invokes
-- Returns:
- ADAG
-
getInvoke
public java.util.List<Invoke> getInvoke()
Returns a list of Invoke objects associated with the workflow- Returns:
-
getNotification
public java.util.List<Invoke> getNotification()
Returns a list of Invoke objects associated with the workflow. Same as getInvoke()- Returns:
-
addFile
public ADAG addFile(File file)
Add a RC File object to the top of the DAX.- Parameters:
file
- File object to be added to the RC section- Returns:
- ADAG
- See Also:
File
-
addFiles
public ADAG addFiles(java.util.List<File> files)
Add Files to the RC Section on top of the DAX- Parameters:
files
- ListList of file objects to be added to the RC Section - Returns:
- ADAG
- See Also:
File
-
getFiles
public java.util.List<File> getFiles()
Returns a list of File objects defined as the inDax Replica Catalog- Returns:
-
addExecutable
public ADAG addExecutable(Executable executable)
Add Executable to the DAX- Parameters:
executable
- Executable to be added- Returns:
- ADAG
- See Also:
Executable
-
addExecutables
public ADAG addExecutables(java.util.List<Executable> executables)
Add Multiple Executable objects to the DAX- Parameters:
executables
- List of Executable objects to be added- Returns:
- ADAG
- See Also:
Executable
-
getExecutables
public java.util.Set<Executable> getExecutables()
Returns a set of Executable Objects stored as part of the inDAX Transformation Catalog;- Returns:
-
containsExecutable
public boolean containsExecutable(Executable executable)
Checks if a given executable exists in the DAX based Transformation Catalog- Parameters:
executable
-- Returns:
- boolean
-
addTransformation
public ADAG addTransformation(Transformation transformation)
Add Transformation to the DAX- Parameters:
transformation
- Transformation object to be added- Returns:
- ADAG
- See Also:
Transformation
-
addTransformations
public ADAG addTransformations(java.util.List<Transformation> transformations)
Add Multiple Transformation to the DAX- Parameters:
transformations
- List of Transformation objects- Returns:
- ADAG
- See Also:
Transformation
-
containsTransformation
public boolean containsTransformation(Transformation transformation)
Checks if a given Transformation exists in the DAX based Transformation Catalog- Parameters:
transformation
- Transformation- Returns:
- boolean
-
getTransformations
public java.util.Set<Transformation> getTransformations()
Returns a set of Transformation Objects (complex executables) stored in the DAX based Transformation Catalog- Returns:
-
addAbstractJob
private ADAG addAbstractJob(AbstractJob ajob)
Add AbstractJob to the DAX- Parameters:
ajob
- AbstractJob- Returns:
- ADAG
- See Also:
Job
,DAG
,DAX
,AbstractJob
-
addAbstractJobs
private ADAG addAbstractJobs(java.util.List<AbstractJob> ajobs)
Add AbstractJobs to the DAX- Parameters:
ajobs
- AbstractJob- Returns:
- ADAG
- See Also:
Job
,DAG
,DAX
,AbstractJob
-
getAbstractJob
private AbstractJob getAbstractJob(java.lang.String ajobid)
Returns an abstract Job with id ajobid if present otherwise null.- Parameters:
ajobid
-- Returns:
-
containsAbstractJob
private boolean containsAbstractJob(AbstractJob ajob)
Check if an abstractjob exists in the DAX- Parameters:
ajob
-- Returns:
-
containsAbstractJobId
private boolean containsAbstractJobId(java.lang.String ajobid)
Check if a jobid exists in the DAX- Parameters:
ajobid
-- Returns:
-
addJob
public ADAG addJob(Job job)
Add Job to the DAX- Parameters:
job
-- Returns:
- ADAG
- See Also:
Job
,AbstractJob
-
addJobs
public ADAG addJobs(java.util.List<Job> jobs)
Add multiple Jobs to the DAX- Parameters:
jobs
-- Returns:
- ADAG
- See Also:
Job
,AbstractJob
-
containsJob
public boolean containsJob(Job job)
Check if a job exists in the DAX- Parameters:
job
-- Returns:
-
containsJobId
public boolean containsJobId(java.lang.String jobid)
Check if a jobid exists in the DAX- Parameters:
jobid
-- Returns:
-
getJob
public Job getJob(java.lang.String jobid)
Returns a Job object with id jobid if present otherwise null.- Parameters:
jobid
-- Returns:
-
getJobs
public java.util.List<Job> getJobs()
Get a list of all the DAG jobs.- Returns:
-
getDAXs
public java.util.List<DAX> getDAXs()
Get a list of all the DAX jobs.- Returns:
-
getDAX
public DAX getDAX(java.lang.String daxid)
Returns a DAX object with id daxid if present otherwise null.- Parameters:
daxid
-- Returns:
-
getDAGs
public java.util.List<DAG> getDAGs()
Get a list of all the DAG jobs.- Returns:
-
getDAG
public DAG getDAG(java.lang.String dagid)
Returns a DAG object with id dagid if present otherwise null.- Parameters:
dagid
-- Returns:
-
addDAG
public ADAG addDAG(DAG dag)
Add a DAG job to the DAX- Parameters:
dag
- the DAG to be added- Returns:
- ADAG
- See Also:
DAG
,AbstractJob
-
addDAGs
public ADAG addDAGs(java.util.List<DAG> dags)
Add multiple DAG jobs to the DAX- Parameters:
dags
- List of DAG jobs to be added- Returns:
- ADAG
- See Also:
DAG
,AbstractJob
-
containsDAG
public boolean containsDAG(DAG dag)
Check if a DAG job exists in the DAX- Parameters:
dag
-- Returns:
-
containsDAGId
public boolean containsDAGId(java.lang.String dagid)
Check if a DAG job id exists in the DAX- Parameters:
dagid
-- Returns:
-
addDAX
public ADAG addDAX(DAX dax)
Add a DAX job to the DAX- Parameters:
dax
- DAX to be added- Returns:
- ADAG
- See Also:
DAX
,AbstractJob
-
addDAXs
public ADAG addDAXs(java.util.List<DAX> daxs)
Add multiple DAX jobs to the DAX- Parameters:
daxs
- LIST of DAX jobs to be added- Returns:
- ADAG
- See Also:
DAX
,AbstractJob
-
containsDAX
public boolean containsDAX(DAX dax)
Check if a DAX job exists in the DAX- Parameters:
dax
-- Returns:
-
containsDAXId
public boolean containsDAXId(java.lang.String daxid)
Check if a DAX job id exists in the DAX- Parameters:
daxid
-- Returns:
-
addDependency
public ADAG addDependency(java.lang.String parent, java.lang.String child)
Add a parent child dependency between two jobs,dax,dag- Parameters:
parent
- String job,dax,dag idchild
- String job,dax,dag,id- Returns:
- ADAG
-
addDependency
public ADAG addDependency(AbstractJob parent, AbstractJob child)
Add a parent child dependency between two jobs,dax,dag- Parameters:
parent
- Job|DAX|DAG objectchild
- Job|DAX|DAG object- Returns:
-
addDependency
public ADAG addDependency(java.lang.String parent, java.lang.String child, java.lang.String label)
Add a parent child dependency with a dependency label- Parameters:
parent
- String job,dax,dag idchild
- String job,dax,dag idlabel
- String dependency label- Returns:
- ADAG
-
getEdges
public java.util.Set<Edge> getEdges(java.lang.String child)
Returns a list of Edge objects for a child job/dax/dag id. Returns an empty set if the child does not have any parents Returns null if the child is not a valid job/dax/dag id- Parameters:
child
-- Returns:
-
getEdges
public java.util.Set<Edge> getEdges()
Returns a Set of all the Edge objects for the DAX. Returns empty if no dependencies.- Parameters:
child
-- Returns:
-
addDependency
public ADAG addDependency(AbstractJob parent, AbstractJob child, java.lang.String label)
Add a parent child dependency with a dependency label- Parameters:
parent
- Job|DAX|DAG objectchild
- Job|DAX|DAG objectlabel
- String label for annotation- Returns:
- ADAG
-
writeToFile
public void writeToFile(java.lang.String daxfile)
Generate a DAX File out of this object;- Parameters:
daxfile
- The file to write the DAX to
-
writeToSTDOUT
public void writeToSTDOUT()
Generate a DAX representation on STDOUT.
-
writeToWriter
public void writeToWriter(java.io.Writer writer, boolean close)
Generate a DAX representation and pipe it into the Writer- Parameters:
writer
- A Writer objectclose
- Whether writer should be closed on return.
-
toXML
public void toXML(XMLWriter writer)
Generates a DAX representation.- Parameters:
writer
-
-
main
public static void main(java.lang.String[] args)
Create an example DIAMOND DAX- Parameters:
args
-
-
Diamond
private static ADAG Diamond()
-
-