Package edu.isi.pegasus.planner.refiner
Class InterPoolEngine
- java.lang.Object
-
- edu.isi.pegasus.planner.refiner.Engine
-
- edu.isi.pegasus.planner.refiner.InterPoolEngine
-
-
Field Summary
Fields Modifier and Type Field Description private ADag
mDag
ADag object corresponding to the Dag whose jobs we want to schedule.protected TransformationStore
mDAXTransformationStore
Handle to the transformation store that stores the transformation catalog user specifies in the DAXprivate java.util.Set
mExecPools
Set of the execution pools which the user has specified.private SiteSelector
mSiteSelector
Handle to the site selector.private SLS
mSLS
The handle to the SLS implementorprivate Mapper
mTCMapper
The handle to the transformation catalog mapper object that caches the queries to the transformation catalog, and indexes them according to lfn's.private TransformationSelector
mTXSelector
The handle to the transformation selector, that ends up selecting what transformations to pick up.private boolean
mWorkerNodeExecution
A boolean indicating whether to have worker node execution or not.private XMLProducer
mXMLStore
The XML Producer object that records the actions.static java.lang.String
REFINER_NAME
The name of the refiner for purposes of error logging-
Fields inherited from class edu.isi.pegasus.planner.refiner.Engine
mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE
-
-
Constructor Summary
Constructors Constructor Description InterPoolEngine(ADag aDag, PegasusBag bag)
Overloaded constructor.InterPoolEngine(PegasusBag bag)
Default constructor.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description java.util.List
convertToList(java.util.Set s)
Converts a Set to a List.java.util.List
convertToList(java.util.Vector v)
Converts a Vector to a List.void
determineSites()
This is where the callout to the Partitioner should take place, that partitions the workflow into clusters and sends to the site selector only those list of jobs that are ready to be scheduled.private java.lang.String
getJobManager(java.lang.String site, java.lang.String universe)
It returns a jobmanager for the given pool.PegasusBag
getPegasusBag()
Returns the bag of intialization objects.java.lang.String
getStagingSite(Job job)
Returns the staging site to be used for a job.ADag
getWorkflow()
Returns a reference to the workflow that is being refined by the refiner.XMLProducer
getXMLProducer()
Returns a reference to the XMLProducer, that generates the XML fragment capturing the actions of the refiner.private void
handleDependantExecutables(Job job)
Handles the dependant executables that need to be staged.private boolean
incorporateHint(Job job, java.lang.String key)
It incorporates a hint in the namespace to the job.private boolean
incorporateProfiles(Job job)
Incorporates the profiles from the various sources into the job.protected void
logRefinerAction(Job job)
Logs the action taken by the refiner on a job as a XML fragment in the XML Producer.void
scheduleJobs(ADag dag, java.util.List sites)
It schedules a list of jobs on the execution pools by calling out to the site selector specified.private TransformationCatalogEntry
selectTCEntry(java.util.List entries, Job job, java.lang.String selector)
Calls out to the transformation selector to select an entry from a list of valid transformation catalog entries.-
Methods inherited from class edu.isi.pegasus.planner.refiner.Engine
addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString
-
-
-
-
Field Detail
-
REFINER_NAME
public static final java.lang.String REFINER_NAME
The name of the refiner for purposes of error logging- See Also:
- Constant Field Values
-
mDag
private ADag mDag
ADag object corresponding to the Dag whose jobs we want to schedule.
-
mExecPools
private java.util.Set mExecPools
Set of the execution pools which the user has specified.
-
mSiteSelector
private SiteSelector mSiteSelector
Handle to the site selector.
-
mTXSelector
private TransformationSelector mTXSelector
The handle to the transformation selector, that ends up selecting what transformations to pick up.
-
mTCMapper
private Mapper mTCMapper
The handle to the transformation catalog mapper object that caches the queries to the transformation catalog, and indexes them according to lfn's. There is no purge policy in the TC Mapper, so per se it is not a classic cache.
-
mXMLStore
private XMLProducer mXMLStore
The XML Producer object that records the actions.
-
mSLS
private SLS mSLS
The handle to the SLS implementor
-
mWorkerNodeExecution
private boolean mWorkerNodeExecution
A boolean indicating whether to have worker node execution or not.
-
mDAXTransformationStore
protected TransformationStore mDAXTransformationStore
Handle to the transformation store that stores the transformation catalog user specifies in the DAX
-
-
Constructor Detail
-
InterPoolEngine
public InterPoolEngine(PegasusBag bag)
Default constructor.- Parameters:
bag
- the bag of initialization objects.
-
InterPoolEngine
public InterPoolEngine(ADag aDag, PegasusBag bag)
Overloaded constructor.- Parameters:
aDag
- theADag
object corresponding to the Dag for which we want to determine on which pools to run the nodes of the Dag.bag
- the bag of initialization objects
-
-
Method Detail
-
getPegasusBag
public PegasusBag getPegasusBag()
Returns the bag of intialization objects.- Returns:
- PegasusBag
-
getWorkflow
public ADag getWorkflow()
Returns a reference to the workflow that is being refined by the refiner.- Specified by:
getWorkflow
in interfaceRefiner
- Returns:
- ADAG object.
-
getXMLProducer
public XMLProducer getXMLProducer()
Returns a reference to the XMLProducer, that generates the XML fragment capturing the actions of the refiner. This is used for provenace purposes.- Specified by:
getXMLProducer
in interfaceRefiner
- Returns:
- XMLProducer
-
determineSites
public void determineSites()
This is where the callout to the Partitioner should take place, that partitions the workflow into clusters and sends to the site selector only those list of jobs that are ready to be scheduled.
-
scheduleJobs
public void scheduleJobs(ADag dag, java.util.List sites)
It schedules a list of jobs on the execution pools by calling out to the site selector specified. It is upto to the site selector to determine if the job can be run on the list of sites passed.- Parameters:
dag
- the abstract workflow.sites
- the list of execution sites, specified by the user.
-
getStagingSite
public java.lang.String getStagingSite(Job job)
Returns the staging site to be used for a job. If a staging site is not determined from the options it is set to be the execution site for the job- Parameters:
job
- the job for which to determine the staging site- Returns:
- the staging site
-
incorporateProfiles
private boolean incorporateProfiles(Job job)
Incorporates the profiles from the various sources into the job. The profiles are incorporated in the order pool, transformation catalog, and properties file, with the profiles from the properties file having the highest priority. It is here where the transformation selector is called to select amongst the various transformations returned by the TC Mapper.- Parameters:
job
- the job into which the profiles have been incorporated.- Returns:
- true profiles were successfully incorporated. false otherwise
-
handleDependantExecutables
private void handleDependantExecutables(Job job)
Handles the dependant executables that need to be staged.- Parameters:
job
- Job
-
selectTCEntry
private TransformationCatalogEntry selectTCEntry(java.util.List entries, Job job, java.lang.String selector)
Calls out to the transformation selector to select an entry from a list of valid transformation catalog entries.- Parameters:
entries
- list ofTransformationCatalogEntry
objects.job
- the job.selectors
- the selector to be called- Returns:
- the selected
TransformationCatalogEntry
object null when transformation selector is unable to select any transformation
-
getJobManager
private java.lang.String getJobManager(java.lang.String site, java.lang.String universe)
It returns a jobmanager for the given pool.- Parameters:
site
- the name of the pool.universe
- the universe for which you need the scheduler on that particular pool.- Returns:
- the jobmanager for that pool and universe. null if not found.
-
incorporateHint
private boolean incorporateHint(Job job, java.lang.String key)
It incorporates a hint in the namespace to the job. After the hint is incorporated the key is deleted from the hint namespace for that job.- Parameters:
job
- the job that needs the hint to be incorporated.key
- the key in the hint namespace.- Returns:
- true the hint was successfully incorporated. false the hint was not set in job or was not successfully incorporated.
-
convertToList
public java.util.List convertToList(java.util.Vector v)
Converts a Vector to a List. It only copies by reference.- Parameters:
v
- Vector- Returns:
- a ArrayList
-
convertToList
public java.util.List convertToList(java.util.Set s)
Converts a Set to a List. It only copies by reference.- Parameters:
s
- Set- Returns:
- a ArrayList
-
logRefinerAction
protected void logRefinerAction(Job job)
Logs the action taken by the refiner on a job as a XML fragment in the XML Producer.- Parameters:
job
- theJob
containing the job that was mapped to a site.
-
-