Package edu.isi.pegasus.planner.refiner
Class ReplicaCatalogBridge
- java.lang.Object
-
- edu.isi.pegasus.planner.refiner.Engine
-
- edu.isi.pegasus.planner.refiner.ReplicaCatalogBridge
-
public class ReplicaCatalogBridge extends Engine
This coordinates the look up to the Replica Location Service, to determine the logical to physical mappings.- Version:
- $Revision$
- Author:
- Karan Vahi, Gaurang Mehta
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
CACHE_READ_ONLY_KEY
The name of the key that disables writing back to the cache file.static java.lang.String
CACHE_REPLICA_CATALOG_IMPLEMENTER
The name of the Replica Catalog Implementer that serves as the source for cache files.static java.lang.String
CACHE_REPLICA_CATALOG_KEY
The name of the source key for Replica Catalog Implementer that serves as cachestatic java.lang.String
DEFAULT_REGISTRATION_CATEGORY_KEY
Default category for registration jobsstatic java.lang.String
DIRECTORY_REPLICA_CATALOG_IMPLEMENTER
The name of the Replica Catalog Implementer that serves as the source for cache files.static java.lang.String
DIRECTORY_REPLICA_CATALOG_KEY
The name of the source key for Replica Catalog Implementer that serves as cacheprivate ReplicaStore
mCacheStore
The replica store in which we store all the results that are queried from the cache replica catalogs.private ADag
mDag
The DAG being worked upon.private ReplicaStore
mDAXReplicaStore
The DAX Replica Store.private boolean
mDefaultTCRCCreated
A boolean indicating whether the attempt to create a default tc entry has happened or not.private TransformationCatalogEntry
mDefaultTCRCEntry
The default tc entry.private ReplicaStore
mDirectoryReplicaStore
The replica store where we store all the results that are queried from the input directory specified at the command line.private ReplicaStore
mInheritedReplicaStore
The inherited Replica Storeprivate boolean
mRCDown
A boolean variable to desingnate whether the RLI queried was down or not.private ReplicaCatalog
mReplicaCatalog
The handle to the main Replica Catalog.private ReplicaStore
mReplicaStore
The replica store in which we store all the results that are queried from the main replica catalog.protected java.util.Set
mSearchFiles
The Vector ofString
objects containing the logical filenames of the files whose locations are to be searched in the Replica Catalog.private boolean
mTreatCacheAsRC
A boolean indicating whether the cache file needs to be treated as a replica catalog or not.static java.lang.String
RC_DERIVATION_NAME
The derivation name for the transfer jobs.static java.lang.String
RC_DERIVATION_NS
The derivation namespace for the transfer jobs.static java.lang.String
RC_DERIVATION_VERSION
The version number for the derivations for registration jobs.static java.lang.String
RC_TRANSFORMATION_NAME
The logical name of the transformation used.static java.lang.String
RC_TRANSFORMATION_NS
The transformation namespace for the regostration jobs.static java.lang.String
RC_TRANSFORMATION_VERSION
The logical name of the transformation used.static java.lang.String
REPLICA_CATALOG_URL_KEY
The name of the URL key for the replica catalog impelementer to be picked up.-
Fields inherited from class edu.isi.pegasus.planner.refiner.Engine
mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE
-
-
Constructor Summary
Constructors Constructor Description ReplicaCatalogBridge(ADag dag, PegasusBag bag)
The overloaded constructor.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
appendProperty(java.lang.StringBuffer sb, java.lang.String key, java.lang.String value)
Appends a property to the StringBuffer, in the java command line format.void
closeConnection()
To close the connection to replica services.private TransformationCatalogEntry
defaultTCRCEntry()
Returns a default TC entry to be used in case entry is not found in the transformation catalog.void
finalize()
Closes the connection to the rli.private java.lang.String
ftToRC(FileTransfer ft)
Converts aFileTransfer
to a RC compatible string representation.private java.lang.String
generateMappingsFile(java.lang.String regJob, java.util.Collection files)
Generates the registration mappings in a text file that is generated in the dax directory (the directory where all the condor submit files are generated).private java.lang.String
generateRepJobArgumentString(SiteCatalogEntry site, java.lang.String regJob, java.util.Collection files)
Generates the argument string to be given to the replica registration job.private Profile
getClassPath(java.lang.String home)
Returns the classpath for the default rc-client entry.protected java.lang.String
getCommandLineProperties(PegasusProperties properties)
Returns the properties that need to be passed to the the rc-client invocation on the command line .java.lang.String
getDefaultRegistrationMaxJobsPropertyKey()
Returns the property key that can be used to set the max jobs for the default category associated with the registration jobs.ReplicaLocation
getFileLocs(java.lang.String lfn)
Returns all the locations as returned from the Replica Lookup Mechanism.java.util.Set
getFilesInReplica()
This returns the files for which mappings exist in the Replica Catalog.private java.lang.String
getPathToLocalProxy()
Returns path to the local proxyprivate ReplicaStore
getReplicaStoreFromDirectory(java.lang.String directory)
Loads the mappings from the input directoryprivate ReplicaStore
getReplicaStoreFromFiles(java.util.Set files)
Ends up loading a Replica Store from replica catalog filesvoid
initialize(ADag dag, PegasusBag bag)
Intialises the refiner.void
initialize(ADag dag, PegasusProperties properties, PlannerOptions options)
Intialises the refiner.private void
loadCacheFiles(java.util.Set cacheFiles)
Ends up loading the cache files so as to enable the lookup for the transient files created by the parent jobs.private void
loadInheritedReplicaStore(java.util.Set files)
Ends up loading the inherited replica files.Job
makeRCRegNode(java.lang.String regJobName, Job job, java.util.Collection files)
It constructs the Job object for the registration node, which registers the materialized files on the output pool in the RLS.private ReplicaLocation
retrieveFromCache(java.lang.String lfn)
Retrieves a location from the cache table, that contains the contents of the cache files specified at runtime.-
Methods inherited from class edu.isi.pegasus.planner.refiner.Engine
addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString
-
-
-
-
Field Detail
-
DEFAULT_REGISTRATION_CATEGORY_KEY
public static final java.lang.String DEFAULT_REGISTRATION_CATEGORY_KEY
Default category for registration jobs- See Also:
- Constant Field Values
-
RC_TRANSFORMATION_NS
public static final java.lang.String RC_TRANSFORMATION_NS
The transformation namespace for the regostration jobs.- See Also:
- Constant Field Values
-
RC_TRANSFORMATION_NAME
public static final java.lang.String RC_TRANSFORMATION_NAME
The logical name of the transformation used.- See Also:
- Constant Field Values
-
RC_TRANSFORMATION_VERSION
public static final java.lang.String RC_TRANSFORMATION_VERSION
The logical name of the transformation used.
-
RC_DERIVATION_NS
public static final java.lang.String RC_DERIVATION_NS
The derivation namespace for the transfer jobs.- See Also:
- Constant Field Values
-
RC_DERIVATION_NAME
public static final java.lang.String RC_DERIVATION_NAME
The derivation name for the transfer jobs.- See Also:
- Constant Field Values
-
RC_DERIVATION_VERSION
public static final java.lang.String RC_DERIVATION_VERSION
The version number for the derivations for registration jobs.- See Also:
- Constant Field Values
-
CACHE_REPLICA_CATALOG_IMPLEMENTER
public static final java.lang.String CACHE_REPLICA_CATALOG_IMPLEMENTER
The name of the Replica Catalog Implementer that serves as the source for cache files.- See Also:
- Constant Field Values
-
CACHE_REPLICA_CATALOG_KEY
public static final java.lang.String CACHE_REPLICA_CATALOG_KEY
The name of the source key for Replica Catalog Implementer that serves as cache- See Also:
- Constant Field Values
-
CACHE_READ_ONLY_KEY
public static final java.lang.String CACHE_READ_ONLY_KEY
The name of the key that disables writing back to the cache file. Designates a static file. i.e. read only- See Also:
- Constant Field Values
-
DIRECTORY_REPLICA_CATALOG_IMPLEMENTER
public static final java.lang.String DIRECTORY_REPLICA_CATALOG_IMPLEMENTER
The name of the Replica Catalog Implementer that serves as the source for cache files.- See Also:
- Constant Field Values
-
DIRECTORY_REPLICA_CATALOG_KEY
public static final java.lang.String DIRECTORY_REPLICA_CATALOG_KEY
The name of the source key for Replica Catalog Implementer that serves as cache- See Also:
- Constant Field Values
-
REPLICA_CATALOG_URL_KEY
public static final java.lang.String REPLICA_CATALOG_URL_KEY
The name of the URL key for the replica catalog impelementer to be picked up.- See Also:
- Constant Field Values
-
mReplicaCatalog
private ReplicaCatalog mReplicaCatalog
The handle to the main Replica Catalog.
-
mSearchFiles
protected java.util.Set mSearchFiles
The Vector ofString
objects containing the logical filenames of the files whose locations are to be searched in the Replica Catalog.
-
mRCDown
private boolean mRCDown
A boolean variable to desingnate whether the RLI queried was down or not. By default it is up, unless it is set to true explicitly.
-
mReplicaStore
private ReplicaStore mReplicaStore
The replica store in which we store all the results that are queried from the main replica catalog.
-
mCacheStore
private ReplicaStore mCacheStore
The replica store in which we store all the results that are queried from the cache replica catalogs.
-
mDirectoryReplicaStore
private ReplicaStore mDirectoryReplicaStore
The replica store where we store all the results that are queried from the input directory specified at the command line.
-
mDAXReplicaStore
private ReplicaStore mDAXReplicaStore
The DAX Replica Store.
-
mInheritedReplicaStore
private ReplicaStore mInheritedReplicaStore
The inherited Replica Store
-
mTreatCacheAsRC
private boolean mTreatCacheAsRC
A boolean indicating whether the cache file needs to be treated as a replica catalog or not.
-
mDefaultTCRCEntry
private TransformationCatalogEntry mDefaultTCRCEntry
The default tc entry.
-
mDefaultTCRCCreated
private boolean mDefaultTCRCCreated
A boolean indicating whether the attempt to create a default tc entry has happened or not.
-
mDag
private ADag mDag
The DAG being worked upon.
-
-
Constructor Detail
-
ReplicaCatalogBridge
public ReplicaCatalogBridge(ADag dag, PegasusBag bag)
The overloaded constructor.- Parameters:
dag
- the workflow that is being worked on.bag
- of initialization objects.
-
-
Method Detail
-
initialize
public void initialize(ADag dag, PegasusBag bag)
Intialises the refiner.- Parameters:
dag
- the workflow that is being worked on.bag
- the bag of Pegasus initialization objects
-
initialize
public void initialize(ADag dag, PegasusProperties properties, PlannerOptions options)
Intialises the refiner.- Parameters:
dag
- the workflow that is being worked on.properties
- the properties passed to the planner.options
- the options passed to the planner at runtime.
-
closeConnection
public void closeConnection()
To close the connection to replica services. This must be defined in the case where one has not done a singleton implementation. In other cases just do an empty implementation of this method.
-
finalize
public void finalize()
Closes the connection to the rli.- Overrides:
finalize
in classjava.lang.Object
-
getFilesInReplica
public java.util.Set getFilesInReplica()
This returns the files for which mappings exist in the Replica Catalog. This should return a subset of the files which are specified in the mSearchFiles, while getting an instance to this.- Returns:
- a
Set
of logical file names as String objects, for which logical to physical mapping exists. - See Also:
mSearchFiles
-
getFileLocs
public ReplicaLocation getFileLocs(java.lang.String lfn)
Returns all the locations as returned from the Replica Lookup Mechanism.- Parameters:
lfn
- The name of the logical file whose PFN mappings are required.- Returns:
- ReplicaLocation containing all the locations for that LFN
- See Also:
org.griphyn.cPlanner.classes.ReplicaLocation
-
getDefaultRegistrationMaxJobsPropertyKey
public java.lang.String getDefaultRegistrationMaxJobsPropertyKey()
Returns the property key that can be used to set the max jobs for the default category associated with the registration jobs.- Returns:
- the property key
-
makeRCRegNode
public Job makeRCRegNode(java.lang.String regJobName, Job job, java.util.Collection files)
It constructs the Job object for the registration node, which registers the materialized files on the output pool in the RLS. Note that the relations corresponding to this node should already have been added to the concernedDagInfo
object.- Parameters:
regJobName
- The name of the job which registers the files in the Replica Location Service.job
- The job whose output files are to be registered in the Replica Location Service.files
- Collection ofFileTransfer
objects containing the information about source and destination URLs. The destination URLs would be our PFNs.- Returns:
- Job corresponding to the new registration node.
-
defaultTCRCEntry
private TransformationCatalogEntry defaultTCRCEntry()
Returns a default TC entry to be used in case entry is not found in the transformation catalog.- Returns:
- the default entry.
-
getClassPath
private Profile getClassPath(java.lang.String home)
Returns the classpath for the default rc-client entry.- Parameters:
home
- the home directory where we need to check for lib directory.- Returns:
- the classpath in an environment profile.
-
generateRepJobArgumentString
private java.lang.String generateRepJobArgumentString(SiteCatalogEntry site, java.lang.String regJob, java.util.Collection files)
Generates the argument string to be given to the replica registration job. At present by default it would be picking up the file containing the mappings.- Parameters:
site
- theSiteCatalogEntry
objectregJob
- The name of the registration job.files
- Collection ofFileTransfer
objects containing the information about source and destURLs. The destination URLs would be our PFNs.- Returns:
- the argument string.
-
getCommandLineProperties
protected java.lang.String getCommandLineProperties(PegasusProperties properties)
Returns the properties that need to be passed to the the rc-client invocation on the command line . It is of the form "-Dprop1=value1 -Dprop2=value2 .."- Parameters:
properties
- the properties object- Returns:
- the properties list, else empty string.
-
appendProperty
protected void appendProperty(java.lang.StringBuffer sb, java.lang.String key, java.lang.String value)
Appends a property to the StringBuffer, in the java command line format.- Parameters:
sb
- the StringBuffer to append the property to.key
- the property.value
- the property value.
-
generateMappingsFile
private java.lang.String generateMappingsFile(java.lang.String regJob, java.util.Collection files)
Generates the registration mappings in a text file that is generated in the dax directory (the directory where all the condor submit files are generated). The pool value for the mapping is the output pool specified by the user when running Pegasus. The name of the file is regJob+.in- Parameters:
regJob
- The name of the registration job.files
- Collection ofFileTransfer
objects containing the information about source and destURLs. The destination URLs would be our PFNs.- Returns:
- String corresponding to the path of the the file containig the mappings in the appropriate format.
-
ftToRC
private java.lang.String ftToRC(FileTransfer ft)
Converts aFileTransfer
to a RC compatible string representation.- Parameters:
ft
- theFileTransfer
object- Returns:
- the RC version.
-
retrieveFromCache
private ReplicaLocation retrieveFromCache(java.lang.String lfn)
Retrieves a location from the cache table, that contains the contents of the cache files specified at runtime.- Parameters:
lfn
- the logical name of the file.- Returns:
ReplicaLocation
object corresponding to the entry if found, else null.
-
loadInheritedReplicaStore
private void loadInheritedReplicaStore(java.util.Set files)
Ends up loading the inherited replica files.- Parameters:
files
- set of paths to the inherited replica files.
-
loadCacheFiles
private void loadCacheFiles(java.util.Set cacheFiles)
Ends up loading the cache files so as to enable the lookup for the transient files created by the parent jobs.- Parameters:
cacheFiles
- set of paths to the cache files.
-
getReplicaStoreFromFiles
private ReplicaStore getReplicaStoreFromFiles(java.util.Set files)
Ends up loading a Replica Store from replica catalog files- Parameters:
files
- set of paths to the cache files.
-
getReplicaStoreFromDirectory
private ReplicaStore getReplicaStoreFromDirectory(java.lang.String directory)
Loads the mappings from the input directory- Parameters:
directory
- the directory to load from
-
getPathToLocalProxy
private java.lang.String getPathToLocalProxy()
Returns path to the local proxy- Returns:
- path to the local proxy
-
-