Package edu.isi.pegasus.planner.refiner

Class InterPoolEngine

  • All Implemented Interfaces:
    Refiner
    public class InterPoolEngine
extends Engine
implements Refiner
    This engine calls out to the Site Selector selected by the user and maps the jobs in the workflow to the execution pools.
    Version:
    $Revision$
    Author:
    Karan Vahi, Gaurang Mehta

    • Field Detail

      • REFINER_NAME

        public static final java.lang.String REFINER_NAME
        The name of the refiner for purposes of error logging
        See Also:
        Constant Field Values

      • mDag

        private ADag mDag
        ADag object corresponding to the Dag whose jobs we want to schedule.

      • mExecPools

        private java.util.Set mExecPools
        Set of the execution pools which the user has specified.

      • mSiteSelector

        private SiteSelector mSiteSelector
        Handle to the site selector.

      • mTXSelector

        private TransformationSelector mTXSelector
        The handle to the transformation selector, that ends up selecting what transformations to pick up.

      • mTCMapper

        private Mapper mTCMapper
        The handle to the transformation catalog mapper object that caches the queries to the transformation catalog, and indexes them according to lfn's. There is no purge policy in the TC Mapper, so per se it is not a classic cache.

      • mXMLStore

        private XMLProducer mXMLStore
        The XML Producer object that records the actions.

      • mSLS

        private SLS mSLS
        The handle to the SLS implementor

      • mWorkerNodeExecution

        private boolean mWorkerNodeExecution
        A boolean indicating whether to have worker node execution or not.

      • mDAXTransformationStore

        protected TransformationStore mDAXTransformationStore
        Handle to the transformation store that stores the transformation catalog user specifies in the DAX

    • Constructor Detail

      • InterPoolEngine

        public InterPoolEngine​(PegasusBag bag)
        Default constructor.
        Parameters:
        bag - the bag of initialization objects.

      • InterPoolEngine

        public InterPoolEngine​(ADag aDag,
                       PegasusBag bag)
        Overloaded constructor.
        Parameters:
        aDag - the ADag object corresponding to the Dag for which we want to determine on which pools to run the nodes of the Dag.
        bag - the bag of initialization objects

    • Method Detail

      • getPegasusBag

        public PegasusBag getPegasusBag()
        Returns the bag of intialization objects.
        Returns:
        PegasusBag

      • getWorkflow

        public ADag getWorkflow()
        Returns a reference to the workflow that is being refined by the refiner.
        Specified by:
        getWorkflow in interface Refiner
        Returns:
        ADAG object.

      • getXMLProducer

        public XMLProducer getXMLProducer()
        Returns a reference to the XMLProducer, that generates the XML fragment capturing the actions of the refiner. This is used for provenace purposes.
        Specified by:
        getXMLProducer in interface Refiner
        Returns:
        XMLProducer

      • determineSites

        public void determineSites()
        This is where the callout to the Partitioner should take place, that partitions the workflow into clusters and sends to the site selector only those list of jobs that are ready to be scheduled.

      • scheduleJobs

        public void scheduleJobs​(ADag dag,
                         java.util.List sites)
        It schedules a list of jobs on the execution pools by calling out to the site selector specified. It is upto to the site selector to determine if the job can be run on the list of sites passed.
        Parameters:
        dag - the abstract workflow.
        sites - the list of execution sites, specified by the user.

      • getStagingSite

        public java.lang.String getStagingSite​(Job job)
        Returns the staging site to be used for a job. If a staging site is not determined from the options it is set to be the execution site for the job
        Parameters:
        job - the job for which to determine the staging site
        Returns:
        the staging site

      • incorporateProfiles

        private boolean incorporateProfiles​(Job job)
        Incorporates the profiles from the various sources into the job. The profiles are incorporated in the order pool, transformation catalog, and properties file, with the profiles from the properties file having the highest priority. It is here where the transformation selector is called to select amongst the various transformations returned by the TC Mapper.
        Parameters:
        job - the job into which the profiles have been incorporated.
        Returns:
        true profiles were successfully incorporated. false otherwise

      • handleDependantExecutables

        private void handleDependantExecutables​(Job job)
        Handles the dependant executables that need to be staged.
        Parameters:
        job - Job

      • selectTCEntry

        private TransformationCatalogEntry selectTCEntry​(java.util.List entries,
                                                 Job job,
                                                 java.lang.String selector)
        Calls out to the transformation selector to select an entry from a list of valid transformation catalog entries.
        Parameters:
        entries - list of TransformationCatalogEntry objects.
        job - the job.
        selectors - the selector to be called
        Returns:
        the selected TransformationCatalogEntry object null when transformation selector is unable to select any transformation

      • getJobManager

        private java.lang.String getJobManager​(java.lang.String site,
                                       java.lang.String universe)
        It returns a jobmanager for the given pool.
        Parameters:
        site - the name of the pool.
        universe - the universe for which you need the scheduler on that particular pool.
        Returns:
        the jobmanager for that pool and universe. null if not found.

      • incorporateHint

        private boolean incorporateHint​(Job job,
                                java.lang.String key)
        It incorporates a hint in the namespace to the job. After the hint is incorporated the key is deleted from the hint namespace for that job.
        Parameters:
        job - the job that needs the hint to be incorporated.
        key - the key in the hint namespace.
        Returns:
        true the hint was successfully incorporated. false the hint was not set in job or was not successfully incorporated.

      • convertToList

        public java.util.List convertToList​(java.util.Vector v)
        Converts a Vector to a List. It only copies by reference.
        Parameters:
        v - Vector
        Returns:
        a ArrayList

      • convertToList

        public java.util.List convertToList​(java.util.Set s)
        Converts a Set to a List. It only copies by reference.
        Parameters:
        s - Set
        Returns:
        a ArrayList

      • logRefinerAction

        protected void logRefinerAction​(Job job)
        Logs the action taken by the refiner on a job as a XML fragment in the XML Producer.
        Parameters:
        job - the Job containing the job that was mapped to a site.