Package edu.isi.pegasus.planner.transfer.sls

Class Transfer

  • All Implemented Interfaces:
    SLS
    public class Transfer
extends java.lang.Object
implements SLS
    This uses the transfer executable distributed with Pegasus to do the second level staging.
    Version:
    $Revision$
    Author:
    Karan Vahi

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static int DEFAULT_NUMBER_OF_THREADS
      The default number of threads pegasus-transfer uses
      static java.lang.String DERIVATION_NAME
      The name of the underlying derivation.
      static java.lang.String DERIVATION_NAMESPACE
      The derivation namespace for for the transfer job.
      static java.lang.String DERIVATION_VERSION
      The derivation version number for the transfer job.
      static java.lang.String DESCRIPTION
      A short description of the transfer implementation.
      static java.lang.String EXECUTABLE_BASENAME
      The executable basename
      private boolean mBypassStagingForInputs
      A boolean indicating whether to bypass first level staging for inputs
      protected java.lang.String mExtraArguments
      Any extra arguments that need to be passed ahead to the s3 client invocation.
      protected LogManager mLogger
      The handle to the logging manager.
      private PlannerCache mPlannerCache
      A SimpleFile Replica Catalog, that tracks all the files that are being materialized as part of workflow execution.
      protected PegasusProperties mProps
      The handle to the properties.
      protected boolean mSeqExecGridStartUsed
      Boolean to track whether the gridstart used in PegasusLite or not
      protected SiteStore mSiteStore
      The handle to the site catalog.
      protected boolean mStageSLSFile
      Boolean to track whether to stage sls file or not
      protected TransformationCatalog mTCHandle
      The handle to the transformation catalog.
      protected boolean mUseSymLinks
      This member variable if set causes the destination URL for the symlink jobs to have symlink:// url if the pool attributed associated with the pfn is same as a particular jobs execution pool.
      static java.lang.String TRANSFORMATION_NAME
      The name of the underlying transformation that is queried for in the Transformation Catalog.
      static java.lang.String TRANSFORMATION_NAMESPACE
      The transformation namespace for the transfer job.
      static java.lang.String TRANSFORMATION_VERSION
      The version number for the transfer job.

      • Fields inherited from interface edu.isi.pegasus.planner.transfer.SLS

        VERSION

    • Constructor Summary

      Constructors 
      Constructor Description
      Transfer()
      The default constructor.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected void complainForHeadNodeURLPrefix​(Job job, java.lang.String site)
      Complains for head node url prefix not specified
      protected void complainForHeadNodeURLPrefix​(java.lang.String site)
      Complains for head node url prefix not specified
      protected TransformationCatalogEntry defaultTCEntry​(java.lang.String namespace, java.lang.String name, java.lang.String version, java.lang.String executableBasename, java.lang.String site)
      Returns a default TC entry to be used in case entry is not found in the transformation catalog.
      java.util.Collection<FileTransfer> determineSLSInputTransfers​(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
      Generates a second level staging file of the input files to the worker node directory.
      java.util.Collection<FileTransfer> determineSLSOutputTransfers​(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)
      Generates a second level staging file of the input files to the worker node directory.
      boolean doesCondorModifications()
      Returns a boolean whether the SLS implementation does a condor based modification or not.
      protected java.util.List getEnvironmentVariables​(java.lang.String site)
      Returns the environment profiles that are required for the default entry to sensibly work.
      protected java.lang.String getExecutableBasename()
      Return the executable basename for transfer executable used.
      java.lang.String getSLSInputLFN​(Job job)
      Returns the LFN of sls input file.
      java.lang.String getSLSOutputLFN​(Job job)
      Returns the LFN of sls output file.
      TransformationCatalogEntry getTransformationCatalogEntry​(java.lang.String siteHandle)
      Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.
      void initialize​(PegasusBag bag)
      Initializes the SLS implementation.
      java.lang.String invocationString​(Job job, java.io.File slsFile)
      Constructs a command line invocation for a job, with a given sls file.
      boolean modifyJobForWorkerNodeExecution​(Job job, java.lang.String stagingSiteURLPrefix, java.lang.String stagingSitedirectory, java.lang.String workerNodeDirectory)
      Modifies a compute job for second level staging.
      boolean needsSLSInputTransfers​(Job job)
      Returns a boolean indicating whether it will an input file for a job to do the transfers.
      boolean needsSLSOutputTransfers​(Job job)
      Returns a boolean indicating whether it will an output file for a job to do the transfers.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • TRANSFORMATION_NAMESPACE

        public static final java.lang.String TRANSFORMATION_NAMESPACE
        The transformation namespace for the transfer job.
        See Also:
        Constant Field Values

      • TRANSFORMATION_NAME

        public static final java.lang.String TRANSFORMATION_NAME
        The name of the underlying transformation that is queried for in the Transformation Catalog.
        See Also:
        Constant Field Values

      • TRANSFORMATION_VERSION

        public static final java.lang.String TRANSFORMATION_VERSION
        The version number for the transfer job.

      • DERIVATION_NAMESPACE

        public static final java.lang.String DERIVATION_NAMESPACE
        The derivation namespace for for the transfer job.
        See Also:
        Constant Field Values

      • DERIVATION_NAME

        public static final java.lang.String DERIVATION_NAME
        The name of the underlying derivation.
        See Also:
        Constant Field Values

      • DERIVATION_VERSION

        public static final java.lang.String DERIVATION_VERSION
        The derivation version number for the transfer job.
        See Also:
        Constant Field Values

      • DEFAULT_NUMBER_OF_THREADS

        public static final int DEFAULT_NUMBER_OF_THREADS
        The default number of threads pegasus-transfer uses
        See Also:
        Constant Field Values

      • DESCRIPTION

        public static final java.lang.String DESCRIPTION
        A short description of the transfer implementation.
        See Also:
        Constant Field Values

      • EXECUTABLE_BASENAME

        public static final java.lang.String EXECUTABLE_BASENAME
        The executable basename
        See Also:
        Constant Field Values

      • mSiteStore

        protected SiteStore mSiteStore
        The handle to the site catalog.

      • mLogger

        protected LogManager mLogger
        The handle to the logging manager.

      • mExtraArguments

        protected java.lang.String mExtraArguments
        Any extra arguments that need to be passed ahead to the s3 client invocation.

      • mStageSLSFile

        protected boolean mStageSLSFile
        Boolean to track whether to stage sls file or not

      • mSeqExecGridStartUsed

        protected boolean mSeqExecGridStartUsed
        Boolean to track whether the gridstart used in PegasusLite or not

      • mBypassStagingForInputs

        private boolean mBypassStagingForInputs
        A boolean indicating whether to bypass first level staging for inputs

      • mPlannerCache

        private PlannerCache mPlannerCache
        A SimpleFile Replica Catalog, that tracks all the files that are being materialized as part of workflow execution.

      • mUseSymLinks

        protected boolean mUseSymLinks
        This member variable if set causes the destination URL for the symlink jobs to have symlink:// url if the pool attributed associated with the pfn is same as a particular jobs execution pool.

    • Constructor Detail

      • Transfer

        public Transfer()
        The default constructor.

    • Method Detail

      • initialize

        public void initialize​(PegasusBag bag)
        Initializes the SLS implementation.
        Specified by:
        initialize in interface SLS
        Parameters:
        bag - the bag of objects. Contains access to catalogs etc.

      • doesCondorModifications

        public boolean doesCondorModifications()
        Returns a boolean whether the SLS implementation does a condor based modification or not. By condor based modification we mean whether it uses condor specific classads to achieve the second level staging or not.
        Specified by:
        doesCondorModifications in interface SLS
        Returns:
        false

      • invocationString

        public java.lang.String invocationString​(Job job,
                                         java.io.File slsFile)
        Constructs a command line invocation for a job, with a given sls file. The SLS maybe null. In the case where SLS impl does not read from a file, it is advised to create a file in generateSLSXXX methods, and then read the file in this function and put it on the command line.
        Specified by:
        invocationString in interface SLS
        Parameters:
        job - the job that is being sls enabled
        slsFile - the slsFile can be null
        Returns:
        invocation string

      • needsSLSInputTransfers

        public boolean needsSLSInputTransfers​(Job job)
        Returns a boolean indicating whether it will an input file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do. Always returns true, as we need to transfer the proxy always.
        Specified by:
        needsSLSInputTransfers in interface SLS
        Parameters:
        job - the job being detected.
        Returns:
        true

      • needsSLSOutputTransfers

        public boolean needsSLSOutputTransfers​(Job job)
        Returns a boolean indicating whether it will an output file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do.
        Specified by:
        needsSLSOutputTransfers in interface SLS
        Parameters:
        job - the job being detected.
        Returns:
        true

      • getSLSInputLFN

        public java.lang.String getSLSInputLFN​(Job job)
        Returns the LFN of sls input file.
        Specified by:
        getSLSInputLFN in interface SLS
        Parameters:
        job - Job
        Returns:
        the name of the sls input file.

      • getSLSOutputLFN

        public java.lang.String getSLSOutputLFN​(Job job)
        Returns the LFN of sls output file.
        Specified by:
        getSLSOutputLFN in interface SLS
        Parameters:
        job - Job
        Returns:
        the name of the sls input file.

      • determineSLSInputTransfers

        public java.util.Collection<FileTransfer> determineSLSInputTransfers​(Job job,
                                                                     java.lang.String fileName,
                                                                     FileServer stagingSiteServer,
                                                                     java.lang.String stagingSiteDirectory,
                                                                     java.lang.String workerNodeDirectory)
        Generates a second level staging file of the input files to the worker node directory.
        Specified by:
        determineSLSInputTransfers in interface SLS
        Parameters:
        job - job for which the file is being created
        fileName - name of the file that needs to be written out.
        stagingSiteServer - the file server on the staging site to be used for retrieval of files i.e the get operation
        stagingSiteDirectory - directory on the head node of the staging site.
        workerNodeDirectory - worker node directory
        Returns:
        a Collection of FileTransfer objects listing the transfers that need to be done.
        See Also:
        needsSLSInputTransfers( Job)

      • determineSLSOutputTransfers

        public java.util.Collection<FileTransfer> determineSLSOutputTransfers​(Job job,
                                                                      java.lang.String fileName,
                                                                      FileServer stagingSiteServer,
                                                                      java.lang.String stagingSiteDirectory,
                                                                      java.lang.String workerNodeDirectory)
        Generates a second level staging file of the input files to the worker node directory.
        Specified by:
        determineSLSOutputTransfers in interface SLS
        Parameters:
        job - the job for which the file is being created
        fileName - the name of the file that needs to be written out.
        stagingSiteServer - the file server on the staging site to be used for retrieval of files i.e the put operation
        stagingSiteDirectory - the directory on the head node of the staging site.
        workerNodeDirectory - the worker node directory
        Returns:
        a Collection of FileTransfer objects listing the transfers that need to be done.
        See Also:
        needsSLSOutputTransfers( Job)

      • modifyJobForWorkerNodeExecution

        public boolean modifyJobForWorkerNodeExecution​(Job job,
                                               java.lang.String stagingSiteURLPrefix,
                                               java.lang.String stagingSitedirectory,
                                               java.lang.String workerNodeDirectory)
        Modifies a compute job for second level staging. The only modification it does is add the appropriate environment varialbes to the job
        Specified by:
        modifyJobForWorkerNodeExecution in interface SLS
        Parameters:
        job - the job to be modified.
        stagingSiteURLPrefix - the url prefix for the server on the staging site
        stagingSitedirectory - the directory on the staging site, where the inp
        workerNodeDirectory - the directory in the worker node tmp
        Returns:
        boolean indicating whether job was successfully modified or not.

      • getTransformationCatalogEntry

        public TransformationCatalogEntry getTransformationCatalogEntry​(java.lang.String siteHandle)
        Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation. If an entry is not specified in the Transformation Catalog, then null is returned.
        Parameters:
        siteHandle - the handle of the site where the transformation is to be searched.
        Returns:
        the transformation catalog entry if found, else null.

      • defaultTCEntry

        protected TransformationCatalogEntry defaultTCEntry​(java.lang.String namespace,
                                                    java.lang.String name,
                                                    java.lang.String version,
                                                    java.lang.String executableBasename,
                                                    java.lang.String site)
        Returns a default TC entry to be used in case entry is not found in the transformation catalog.
        Parameters:
        namespace - the namespace of the transfer transformation
        name - the logical name of the transfer transformation
        version - the version of the transfer transformation
        executableBasename - the basename of the executable
        site - the site for which the default entry is required.
        Returns:
        the default entry.

      • getEnvironmentVariables

        protected java.util.List getEnvironmentVariables​(java.lang.String site)
        Returns the environment profiles that are required for the default entry to sensibly work. Tries to retrieve the following variables 
         PEGASUS_HOME
 GLOBUS_LOCATION
 LD_LIBRARY_PATH
        Parameters:
        site - the site where the job is going to run.
        Returns:
        List of environment variables, else empty list if none are found

      • getExecutableBasename

        protected java.lang.String getExecutableBasename()
        Return the executable basename for transfer executable used.
        Returns:
        the executable basename.

      • complainForHeadNodeURLPrefix

        protected void complainForHeadNodeURLPrefix​(java.lang.String site)
        Complains for head node url prefix not specified
        Parameters:
        site - the site handle
        Throws:
        java.lang.RuntimeException - when URL Prefix cannot be determined for various reason.

      • complainForHeadNodeURLPrefix

        protected void complainForHeadNodeURLPrefix​(Job job,
                                            java.lang.String site)
        Complains for head node url prefix not specified
        Parameters:
        job - the related job if any
        site - the site handle
        Throws:
        java.lang.RuntimeException - when URL Prefix cannot be determined for various reason.