Package edu.isi.pegasus.planner.refiner
Class RemoveDirectory
- java.lang.Object
-
- edu.isi.pegasus.planner.refiner.Engine
-
- edu.isi.pegasus.planner.refiner.RemoveDirectory
-
public class RemoveDirectory extends Engine
Ends up creating a cleanup dag that deletes the remote directories that were created by the create dir jobs. The cleanup dag is generated in a sub directory from the main directory containing the submit files of the dag. The dag consists of independant jobs, with each job responsible for deleting directory for a execution pool. The current way of generating the dag is tied to the fact, that the directories in which a job are executed is tied to the pool not the job itself.- Version:
- $Revision$
- Author:
- Karan Vahi
- See Also:
CreateDirectory
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
CLEANUP_DAG_PREFIX
The prefix that is attached to the name of the dag for which the cleanup Dag is being generated, to generate the name of the cleanup Dag.static java.lang.String
CLEANUP_PREFIX
Constant suffix for the names of the deployment nodes.static java.lang.String
DERIVATION_NAME
The logical name of the transformation that removes directories on the remote execution pools.static java.lang.String
DERIVATION_NAMESPACE
The derivation namespace for the create dir jobs.static java.lang.String
DERIVATION_VERSION
The version number for the derivations for create dir jobs.private ADag
mConcDag
The concrete dag so far, for which the clean up dag needs to be generated.protected java.lang.String
mJobPrefix
The job prefix that needs to be applied to the job file basenames.private java.lang.String
mSubmitDirectory
The submit directory for the workflow.private boolean
mTransferFromSubmitHost
Boolean indicating whether we need to transfer dirmanager from the submit host.static java.lang.String
REMOVE_DIR_EXECUTABLE_BASENAME
The basename of the pegasus dirmanager executable.static java.lang.String
REMOVE_DIR_SUFFIX
Constant suffix for the names of the remote directory nodes.static java.lang.String
TRANSFORMATION_NAME
The logical name of the transformation that removes directories on the remote execution pools.static java.lang.String
TRANSFORMATION_NAMESPACE
The transformation namespace for the create dir jobs.static java.lang.String
TRANSFORMATION_VERSION
The version number for the derivations for create dir jobs.-
Fields inherited from class edu.isi.pegasus.planner.refiner.Engine
mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE
-
-
Constructor Summary
Constructors Constructor Description RemoveDirectory(ADag concDag, PegasusBag bag, java.lang.String submitDirectory)
The overloaded constructor that sets the dag for which we have to generated the cleanup dag for.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description ADag
addRemoveDirectoryNodes(ADag dag)
Modifies the workflow to add remove directory nodes.ADag
addRemoveDirectoryNodes(ADag workflow, java.util.Set<java.lang.String> sites)
Adds create dir nodes to the workflow.private TransformationCatalogEntry
defaultTCEntry(SiteCatalogEntry site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.private java.lang.String
getAssociatedCreateDirSite(Job job)
Returns the associated site that job is dependant on.static java.lang.String
getCompleteTranformationName()
A convenience method to return the complete transformation name being used to construct jobs in this class.protected java.util.Set
getCreateDirSites(ADag dag)
Retrieves the sites for which the create dir jobs need to be created.private java.lang.String
getRemoveDirJobName(ADag dag, java.lang.String site)
It returns the name of the remove directory job, that is to be assigned.Job
makeRemoveDirJob(java.lang.String site, java.lang.String jobName)
It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote.Job
makeRemoveDirJob(java.lang.String site, java.lang.String jobName, java.util.List<java.lang.String> files)
It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote.-
Methods inherited from class edu.isi.pegasus.planner.refiner.Engine
addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString
-
-
-
-
Field Detail
-
CLEANUP_DAG_PREFIX
public static final java.lang.String CLEANUP_DAG_PREFIX
The prefix that is attached to the name of the dag for which the cleanup Dag is being generated, to generate the name of the cleanup Dag.- See Also:
- Constant Field Values
-
REMOVE_DIR_SUFFIX
public static final java.lang.String REMOVE_DIR_SUFFIX
Constant suffix for the names of the remote directory nodes.- See Also:
- Constant Field Values
-
TRANSFORMATION_NAME
public static final java.lang.String TRANSFORMATION_NAME
The logical name of the transformation that removes directories on the remote execution pools.- See Also:
- Constant Field Values
-
REMOVE_DIR_EXECUTABLE_BASENAME
public static final java.lang.String REMOVE_DIR_EXECUTABLE_BASENAME
The basename of the pegasus dirmanager executable.- See Also:
- Constant Field Values
-
TRANSFORMATION_NAMESPACE
public static final java.lang.String TRANSFORMATION_NAMESPACE
The transformation namespace for the create dir jobs.- See Also:
- Constant Field Values
-
TRANSFORMATION_VERSION
public static final java.lang.String TRANSFORMATION_VERSION
The version number for the derivations for create dir jobs.
-
DERIVATION_NAMESPACE
public static final java.lang.String DERIVATION_NAMESPACE
The derivation namespace for the create dir jobs.- See Also:
- Constant Field Values
-
DERIVATION_NAME
public static final java.lang.String DERIVATION_NAME
The logical name of the transformation that removes directories on the remote execution pools.- See Also:
- Constant Field Values
-
DERIVATION_VERSION
public static final java.lang.String DERIVATION_VERSION
The version number for the derivations for create dir jobs.- See Also:
- Constant Field Values
-
CLEANUP_PREFIX
public static final java.lang.String CLEANUP_PREFIX
Constant suffix for the names of the deployment nodes.- See Also:
- Constant Field Values
-
mConcDag
private ADag mConcDag
The concrete dag so far, for which the clean up dag needs to be generated.
-
mTransferFromSubmitHost
private boolean mTransferFromSubmitHost
Boolean indicating whether we need to transfer dirmanager from the submit host.
-
mSubmitDirectory
private java.lang.String mSubmitDirectory
The submit directory for the workflow.
-
mJobPrefix
protected java.lang.String mJobPrefix
The job prefix that needs to be applied to the job file basenames.
-
-
Constructor Detail
-
RemoveDirectory
public RemoveDirectory(ADag concDag, PegasusBag bag, java.lang.String submitDirectory)
The overloaded constructor that sets the dag for which we have to generated the cleanup dag for.- Parameters:
concDag
- the concrete dag for which cleanup is reqd.bag
- the bag of initialization objectssubmitDirectory
- the submit directory for the cleanup workflow
-
-
Method Detail
-
getCompleteTranformationName
public static java.lang.String getCompleteTranformationName()
A convenience method to return the complete transformation name being used to construct jobs in this class.- Returns:
- the complete transformation name
-
addRemoveDirectoryNodes
public ADag addRemoveDirectoryNodes(ADag dag)
Modifies the workflow to add remove directory nodes. The workflow passed is a worklow, where the jobs have been mapped to sites. The strategy involves in walking the graph in a BFS order, and updating a bit set associated with each job based on the BitSet of the parents jobs. The BitSet indicates whether an edge exists from the descendant of a node to the remove dir job. For a node, the bit set is the union of all the parents BitSets. The BFS traversal ensures that the bitsets are of a node are only updated once the parents have been processed.- Parameters:
dag
- the workflow to which the nodes have to be added.- Returns:
- the added workflow
-
addRemoveDirectoryNodes
public ADag addRemoveDirectoryNodes(ADag workflow, java.util.Set<java.lang.String> sites)
Adds create dir nodes to the workflow. The strategy involves in walking the graph in a BFS order, and updating a bit set associated with each job based on the BitSet of the parent jobs. The BitSet indicates whether an edge exists from the create dir job to an ancestor of the node. For a node, the bit set is the union of all the parents BitSets. The BFS traversal ensures that the bitsets are of a node are only updated once the parents have been processed.- Parameters:
workflow
- the workflowsites
- the staging sites the workflow refers to.- Returns:
-
getCreateDirSites
protected java.util.Set getCreateDirSites(ADag dag)
Retrieves the sites for which the create dir jobs need to be created. It returns all the sites where the compute jobs have been scheduled.- Returns:
- a Set containing a list of siteID's of the sites where the dag has to be run.
-
getRemoveDirJobName
private java.lang.String getRemoveDirJobName(ADag dag, java.lang.String site)
It returns the name of the remove directory job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.- Parameters:
dag
- the dag for which the cleanup DAG is being generated.site
- the execution site for which the remove directory job is responsible.- Returns:
- String corresponding to the name of the job.
-
makeRemoveDirJob
public Job makeRemoveDirJob(java.lang.String site, java.lang.String jobName)
It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote. It access mkdir underneath. It gets the name of the random directory from the Pool handle.- Parameters:
site
- the execution pool for which the create dir job is to be created.jobName
- the name that is to be assigned to the job.- Returns:
- the remove dir job.
-
makeRemoveDirJob
public Job makeRemoveDirJob(java.lang.String site, java.lang.String jobName, java.util.List<java.lang.String> files)
It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote. It access mkdir underneath. It gets the name of the random directory from the Pool handle.- Parameters:
site
- the site from where the directory need to be removed.jobName
- the name that is to be assigned to the job.files
- the list of files to be cleaned up.- Returns:
- the remove dir job.
-
defaultTCEntry
private TransformationCatalogEntry defaultTCEntry(SiteCatalogEntry site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.- Parameters:
site
- the SiteCatalogEntry for the site for which the default entry is required.- Returns:
- the default entry.
-
getAssociatedCreateDirSite
private java.lang.String getAssociatedCreateDirSite(Job job)
Returns the associated site that job is dependant on. This is site, whose create dir job should be a parent or an ancestor of the job.- Parameters:
job
- the job for which we need the associated create dir site.- Returns:
- the site
-
-