Class Abstract
- java.lang.Object
-
- edu.isi.pegasus.planner.cluster.aggregator.Abstract
-
- All Implemented Interfaces:
JobAggregator
public abstract class Abstract extends java.lang.Object implements JobAggregator
An abstract implementation of the JobAggregator interface, which the other implementations can choose to extend.- Version:
- $Revision$
- Author:
- Karan Vahi vahi@isi.edu
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
CLUSTERED_JOB_PREFIX
The prefix that is assigned to the jobname of the collapsed jobs to get the jobname for the fat job.static java.lang.String
DERIVATION_NAMESPACE
The derivation namespace for the cluster jobs.static java.lang.String
DERIVATION_VERSION
The version number for the derivations for cluster jobs.protected PegasusBag
mBag
Bag of initialization objects.protected ADag
mClusteredADag
The handle to the ADag object that contains the workflow being clustered.protected java.lang.String
mDirectory
The directory, where the stdin file of the fat jobs are created.protected GridStartFactory
mGridStartFactory
The handle to the GridStart Factory.protected LogManager
mLogger
The handle to the LogManager that logs all the messages.static java.lang.String
MONITORD_COMMENT_MARKER
The marker to designate a line in the input file reserved for monitord purposes.protected PegasusProperties
mProps
The object holding all the properties pertaining to Pegasus.protected SiteStore
mSiteStore
Handle to the site catalog storeprotected TransformationCatalog
mTCHandle
The handle to the transformation catalog.static java.lang.String
TRANSFORMATION_NAMESPACE
The transformation namespace for the cluster jobs.static java.lang.String
TRANSFORMATION_VERSION
The version number for the derivations for cluster jobs-
Fields inherited from interface edu.isi.pegasus.planner.cluster.JobAggregator
VERSION
-
-
Constructor Summary
Constructors Constructor Description Abstract()
The default constructor.
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description abstract java.lang.String
aggregatedJobArguments(AggregatedJob job)
Returns the arguments with which theAggregatedJob
needs to be invoked with.AggregatedJob
constructAbstractAggregatedJob(java.util.List jobs, java.lang.String name, java.lang.String id)
Constructs a new aggregated job that contains all the jobs passed to it.AggregatedJob
constructAbstractAggregatedJob(java.util.List jobs, java.lang.String name, java.lang.String id, java.lang.String mergeLFN)
Constructs an abstract aggregated job that has a handle to the appropriate JobAggregator that will be used to aggregate the jobs.private TransformationCatalogEntry
defaultTCEntry(java.lang.String name, java.lang.String executableBasename, java.lang.String site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.protected boolean
entryNotInTC(java.lang.String namespace, java.lang.String name, java.lang.String version, java.lang.String executableBasename, java.lang.String site)
Determines whether there is NOT an entry in the transformation catalog for a particular transformation on a particular site.protected java.lang.String
getCommentString(Job job, int taskid)
Generates the comment string for the job .static java.lang.String
getCompleteTranformationName(java.lang.String name)
A convenience method to return the complete transformation name being used to construct jobs in this class.protected TransformationCatalogEntry
getTCEntry(Job job)
Helper method to get an entry from the transformation catalog for an installed executable.void
initialize(ADag dag, PegasusBag bag)
Initializes the JobAggregator impelementationvoid
makeAbstractAggregatedJobConcrete(AggregatedJob job)
Enables the abstract clustered job for execution and converts it to it's executable formprotected void
setDirectory(java.lang.String directory)
Sets the directory where the stdin files are to be generated.protected java.io.File
writeOutInputFileForJobAggregator(AggregatedJob job)
Writes out the input file for the aggregated job-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface edu.isi.pegasus.planner.cluster.JobAggregator
abortOnFristJobFailure, entryNotInTC, getClusterExecutableBasename, getClusterExecutableLFN, setAbortOnFirstJobFailure, topologicalOrderingRequired
-
-
-
-
Field Detail
-
CLUSTERED_JOB_PREFIX
public static final java.lang.String CLUSTERED_JOB_PREFIX
The prefix that is assigned to the jobname of the collapsed jobs to get the jobname for the fat job.- See Also:
- Constant Field Values
-
TRANSFORMATION_NAMESPACE
public static final java.lang.String TRANSFORMATION_NAMESPACE
The transformation namespace for the cluster jobs.- See Also:
- Constant Field Values
-
TRANSFORMATION_VERSION
public static final java.lang.String TRANSFORMATION_VERSION
The version number for the derivations for cluster jobs
-
DERIVATION_NAMESPACE
public static final java.lang.String DERIVATION_NAMESPACE
The derivation namespace for the cluster jobs.- See Also:
- Constant Field Values
-
DERIVATION_VERSION
public static final java.lang.String DERIVATION_VERSION
The version number for the derivations for cluster jobs.- See Also:
- Constant Field Values
-
MONITORD_COMMENT_MARKER
public static final java.lang.String MONITORD_COMMENT_MARKER
The marker to designate a line in the input file reserved for monitord purposes.- See Also:
- Constant Field Values
-
mDirectory
protected java.lang.String mDirectory
The directory, where the stdin file of the fat jobs are created. It should be the submit file directory that the user mentions at runtime.
-
mProps
protected PegasusProperties mProps
The object holding all the properties pertaining to Pegasus.
-
mLogger
protected LogManager mLogger
The handle to the LogManager that logs all the messages.
-
mTCHandle
protected TransformationCatalog mTCHandle
The handle to the transformation catalog.
-
mSiteStore
protected SiteStore mSiteStore
Handle to the site catalog store
-
mClusteredADag
protected ADag mClusteredADag
The handle to the ADag object that contains the workflow being clustered.
-
mGridStartFactory
protected GridStartFactory mGridStartFactory
The handle to the GridStart Factory.
-
mBag
protected PegasusBag mBag
Bag of initialization objects.
-
-
Method Detail
-
getCompleteTranformationName
public static java.lang.String getCompleteTranformationName(java.lang.String name)
A convenience method to return the complete transformation name being used to construct jobs in this class.- Parameters:
name
- the name of the transformation- Returns:
- the complete transformation name
-
initialize
public void initialize(ADag dag, PegasusBag bag)
Initializes the JobAggregator impelementation- Specified by:
initialize
in interfaceJobAggregator
- Parameters:
dag
- the workflow that is being clustered.bag
- the bag of objects that is useful for initialization.
-
aggregatedJobArguments
public abstract java.lang.String aggregatedJobArguments(AggregatedJob job)
Returns the arguments with which theAggregatedJob
needs to be invoked with.- Parameters:
job
- theAggregatedJob
for which the arguments have to be constructed.- Returns:
- argument string
-
constructAbstractAggregatedJob
public AggregatedJob constructAbstractAggregatedJob(java.util.List jobs, java.lang.String name, java.lang.String id)
Constructs a new aggregated job that contains all the jobs passed to it. The new aggregated job, appears as a single job in the workflow and replaces the jobs it contains in the workflow.- Specified by:
constructAbstractAggregatedJob
in interfaceJobAggregator
- Parameters:
jobs
- the list ofJob
objects that need to be collapsed. All the jobs being collapsed should be scheduled at the same pool, to maintain correct semantics.name
- the logical name of the jobs in the list passed to this function.id
- the id that is given to the new job.- Returns:
- the
Job
object corresponding to the aggregated job containing the jobs passed as List in the input, null if the list of jobs is empty
-
constructAbstractAggregatedJob
public AggregatedJob constructAbstractAggregatedJob(java.util.List jobs, java.lang.String name, java.lang.String id, java.lang.String mergeLFN)
Constructs an abstract aggregated job that has a handle to the appropriate JobAggregator that will be used to aggregate the jobs.- Parameters:
jobs
- the list ofSubInfo
objects that need to be collapsed. All the jobs being collapsed should be scheduled at the same pool, to maintain correct semantics.name
- the logical name of the jobs in the list passed to this function.id
- the id that is given to the new job.mergeLFN
- the logical name for the aggregated job that has to be constructed.- Returns:
- the
SubInfo
object corresponding to the aggregated job containing the jobs passed as List in the input, null if the list of jobs is empty
-
makeAbstractAggregatedJobConcrete
public void makeAbstractAggregatedJobConcrete(AggregatedJob job)
Enables the abstract clustered job for execution and converts it to it's executable form- Specified by:
makeAbstractAggregatedJobConcrete
in interfaceJobAggregator
- Parameters:
job
- the abstract clustered job
-
getCommentString
protected java.lang.String getCommentString(Job job, int taskid)
Generates the comment string for the job . It generates a comment of the format # task_id transformation derivation.- Parameters:
job
- the job for whichtaskid
- the task id to put in.- Returns:
- the comment invocation
-
getTCEntry
protected TransformationCatalogEntry getTCEntry(Job job)
Helper method to get an entry from the transformation catalog for an installed executable. It does the traversal from the list of entries to return a single TransformationCatalogEntry object, and dies with an appropriate error message if the object is not found. The pool and the name are retrieved from job object.- Parameters:
job
- the job whose corresponding TransformationCatalogEntry you want.- Returns:
- the TransformationCatalogEntry corresponding to the entry in the TC.
-
defaultTCEntry
private TransformationCatalogEntry defaultTCEntry(java.lang.String name, java.lang.String executableBasename, java.lang.String site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.- Parameters:
name
- the logical name for the clustering transformation.executableBasename
- the basename for the executable in the bin directory of a Pegasus installationsite
- the site for which the default entry is required.- Returns:
- the default entry.
-
entryNotInTC
protected boolean entryNotInTC(java.lang.String namespace, java.lang.String name, java.lang.String version, java.lang.String executableBasename, java.lang.String site)
Determines whether there is NOT an entry in the transformation catalog for a particular transformation on a particular site.- Parameters:
namespace
- the logical namespace of the transformation.name
- the logical name of the transformation.version
- the version of the transformation.executableBasename
- basename of the executable that does the clustering.site
- the site at which existence check is required.- Returns:
- boolean true if an entry does not exists, false otherwise.
-
setDirectory
protected void setDirectory(java.lang.String directory)
Sets the directory where the stdin files are to be generated.- Parameters:
directory
- the path to the directory to which it needs to be set.
-
writeOutInputFileForJobAggregator
protected java.io.File writeOutInputFileForJobAggregator(AggregatedJob job)
Writes out the input file for the aggregated job- Parameters:
job
- the aggregated job- Returns:
- path to the input file
-
-