Class Condor

  • All Implemented Interfaces:
    SLS

    public class Condor
    extends java.lang.Object
    implements SLS
    This uses the Condor File Transfer mechanism for the second level staging. It will work only if the Pegasus Style profile ( pegasus::style ) has a value of condor.
    Version:
    $Revision$
    Author:
    Karan Vahi
    • Constructor Summary

      Constructors 
      Constructor Description
      Condor()
      The default constructor.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.util.Collection<FileTransfer> determineSLSInputTransfers​(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
      Generates a second level staging file of the input files to the worker node directory.
      java.util.Collection<FileTransfer> determineSLSOutputTransfers​(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
      Generates a second level staging file of the input files to the worker node directory.
      boolean doesCondorModifications()
      Returns a boolean whether the SLS implementation does a condor based modification or not.
      java.lang.String getSLSInputLFN​(Job job)
      Returns the LFN of sls input file.
      java.lang.String getSLSOutputLFN​(Job job)
      Returns the LFN of sls output file.
      void initialize​(PegasusBag bag)
      Initializes the SLS implementation.
      java.lang.String invocationString​(Job job, java.io.File slsFile)
      Constructs a command line invocation for a job, with a given sls file.
      boolean modifyJobForFirstLevelStaging​(Job job, java.lang.String submitDir, java.lang.String slsInputLFN, java.lang.String slsOutputLFN)
      Modifies a job for the first level staging to headnode.This is to add any files that needs to be staged to the head node for a job specific to the SLS implementation.
      boolean modifyJobForWorkerNodeExecution​(Job job, java.lang.String stagingSiteURLPrefix, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
      Modifies a compute job for second level staging.
      boolean needsSLSInputTransfers​(Job job)
      Returns a boolean indicating whether it will an input file for a job to do the transfers.
      boolean needsSLSOutputTransfers​(Job job)
      Returns a boolean indicating whether it will an output file for a job to do the transfers.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • DESCRIPTION

        public static final java.lang.String DESCRIPTION
        A short description of the transfer implementation.
        See Also:
        Constant Field Values
      • mSiteStore

        protected SiteStore mSiteStore
        The handle to the site catalog.
      • mPOptions

        protected PlannerOptions mPOptions
        The handle to the planner options.
      • mLogger

        protected LogManager mLogger
        The handle to the logging manager.
      • mBypassStagingForInputs

        private boolean mBypassStagingForInputs
        A boolean indicating whether to bypass first level staging for inputs
      • mPlannerCache

        private PlannerCache mPlannerCache
        A SimpleFile Replica Catalog, that tracks all the files that are being materialized as part of workflow execution.
    • Constructor Detail

      • Condor

        public Condor()
        The default constructor.
    • Method Detail

      • initialize

        public void initialize​(PegasusBag bag)
        Initializes the SLS implementation.
        Specified by:
        initialize in interface SLS
        Parameters:
        bag - the bag of objects. Contains access to catalogs etc.
      • doesCondorModifications

        public boolean doesCondorModifications()
        Returns a boolean whether the SLS implementation does a condor based modification or not. By condor based modification we mean whether it uses condor specific classads to achieve the second level staging or not.
        Specified by:
        doesCondorModifications in interface SLS
        Returns:
        false
      • invocationString

        public java.lang.String invocationString​(Job job,
                                                 java.io.File slsFile)
        Constructs a command line invocation for a job, with a given sls file. The SLS maybe null. In the case where SLS impl does not read from a file, it is advised to create a file in generateSLSXXX methods, and then read the file in this function and put it on the command line.
        Specified by:
        invocationString in interface SLS
        Parameters:
        job - the job that is being sls enabled
        slsFile - the slsFile can be null
        Returns:
        invocation string
      • needsSLSInputTransfers

        public boolean needsSLSInputTransfers​(Job job)
        Returns a boolean indicating whether it will an input file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do. Always returns true, as we need to transfer the proxy always.
        Specified by:
        needsSLSInputTransfers in interface SLS
        Parameters:
        job - the job being detected.
        Returns:
        false
      • needsSLSOutputTransfers

        public boolean needsSLSOutputTransfers​(Job job)
        Returns a boolean indicating whether it will an output file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do.
        Specified by:
        needsSLSOutputTransfers in interface SLS
        Parameters:
        job - the job being detected.
        Returns:
        false
      • getSLSInputLFN

        public java.lang.String getSLSInputLFN​(Job job)
        Returns the LFN of sls input file.
        Specified by:
        getSLSInputLFN in interface SLS
        Parameters:
        job - Job
        Returns:
        the name of the sls input file.
      • getSLSOutputLFN

        public java.lang.String getSLSOutputLFN​(Job job)
        Returns the LFN of sls output file.
        Specified by:
        getSLSOutputLFN in interface SLS
        Parameters:
        job - Job
        Returns:
        the name of the sls input file.
      • determineSLSInputTransfers

        public java.util.Collection<FileTransfer> determineSLSInputTransfers​(Job job,
                                                                             java.lang.String fileName,
                                                                             FileServer stagingSiteServer,
                                                                             java.lang.String stagingSiteDirectory,
                                                                             java.lang.String workerNodeDirectory)
        Generates a second level staging file of the input files to the worker node directory.
        Specified by:
        determineSLSInputTransfers in interface SLS
        Parameters:
        job - job for which the file is being created
        fileName - name of the file that needs to be written out.
        stagingSiteServer - the file server on the staging site to be used for retrieval of files i.e the get operation
        stagingSiteDirectory - directory on the head node of the compute site.
        workerNodeDirectory - worker node directory
        Returns:
        a Collection of FileTransfer objects listing the transfers that need to be done.
        See Also:
        needsSLSInputTransfers( Job)
      • determineSLSOutputTransfers

        public java.util.Collection<FileTransfer> determineSLSOutputTransfers​(Job job,
                                                                              java.lang.String fileName,
                                                                              FileServer stagingSiteServer,
                                                                              java.lang.String stagingSiteDirectory,
                                                                              java.lang.String workerNodeDirectory)
        Generates a second level staging file of the input files to the worker node directory.
        Specified by:
        determineSLSOutputTransfers in interface SLS
        Parameters:
        job - the job for which the file is being created
        fileName - the name of the file that needs to be written out.
        stagingSiteServer - the file server on the staging site to be used for retrieval of files i.e the put operation
        stagingSiteDirectory - the directory on the head node of the compute site.
        workerNodeDirectory - the worker node directory
        Returns:
        a Collection of FileTransfer objects listing the transfers that need to be done.
        See Also:
        needsSLSOutputTransfers( Job)
      • modifyJobForFirstLevelStaging

        public boolean modifyJobForFirstLevelStaging​(Job job,
                                                     java.lang.String submitDir,
                                                     java.lang.String slsInputLFN,
                                                     java.lang.String slsOutputLFN)
        Modifies a job for the first level staging to headnode.This is to add any files that needs to be staged to the head node for a job specific to the SLS implementation. If any file needs to be added, a FileTransfer object should be created and added as an input or an output file.
        Parameters:
        job - the job
        submitDir - the submit directory
        slsInputLFN - the sls input file if required, that is used for staging in from the head node to worker node directory.
        slsOutputLFN - the sls output file if required, that is used for staging in from the head node to worker node directory.
        Returns:
        boolean
      • modifyJobForWorkerNodeExecution

        public boolean modifyJobForWorkerNodeExecution​(Job job,
                                                       java.lang.String stagingSiteURLPrefix,
                                                       java.lang.String stagingSiteDirectory,
                                                       java.lang.String workerNodeDirectory)
        Modifies a compute job for second level staging. Adds the appropriate condor classads. It assumes that all the files are being moved to and from the submit directory directly. Ignores any headnode parameters passed.
        Specified by:
        modifyJobForWorkerNodeExecution in interface SLS
        Parameters:
        job - the job to be modified.
        stagingSiteURLPrefix - the url prefix for the server on the staging site
        stagingSitedirectory - the directory on the staging site, where the nput data is read from and the output data written out.
        workerNodeDirectory - the directory in the worker node tmp
        Returns:
        boolean indicating whether job was successfully modified or not.