Class CondorGenerator
- java.lang.Object
-
- edu.isi.pegasus.planner.code.generator.Abstract
-
- edu.isi.pegasus.planner.code.generator.condor.CondorGenerator
-
- All Implemented Interfaces:
CodeGenerator
- Direct Known Subclasses:
HashedFile
public class CondorGenerator extends Abstract
This class generates the condor submit files for the DAG which has to be submitted to the Condor DagMan.- Version:
- $Revision$
- Author:
- Gaurang Mehta, Karan Vahi
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.StringCONDOR_DAGMAN_LOGICAL_NAMEThe logical name with which to query the transformation catalog for the condor_dagman executable, that ends up running the mini dag as one job.static java.lang.StringCONDOR_DAGMAN_NAMESPACEThe namespace to use for condor dagman.static java.lang.StringDAGMAN_PROPERTIES_PREFIXThe prefix for DAGMan specific propertiesstatic intDEFAULT_CHMOD_PRIORITY_KEYThe default priority key associated with chmod jobs.static intDEFAULT_CLEANUP_PRIORITY_KEYThe default priority key associated with the cleanup jobs.static intDEFAULT_CREATE_DIR_PRIORITY_KEYThe default priority key associated with the create dir jobs.static intDEFAULT_INTER_SITE_PRIORITY_KEYThe default priority key associated with the inter site transfer jobs.static java.lang.StringDEFAULT_PERIODIC_RELEASE_VALUEDefault value for the periodic_release for a jobstatic java.lang.StringDEFAULT_PERIODIC_REMOVE_VALUEDefault value for the periodic_remove for a jobstatic intDEFAULT_REPLICA_REG_PRIORITY_KEYThe default priority key associated with the replica registration jobs.static intDEFAULT_STAGE_IN_PRIORITY_KEYThe default priority key associated with the stagein jobs.static intDEFAULT_STAGE_OUT_PRIORITY_KEYThe default priority key associated with the stage out jobs.static java.lang.StringDEFAULT_SUBDAG_CATEGORY_KEYThe default category for the sub dax jobs.private booleanmAssignDefaultJobPrioritiesBoolean indicating whether to assign job priorities or not.protected ADagmConcreteWorkflowThe workflow for which the code has to be generated.private longmCondorVersionThe long value of condor version.protected java.io.PrintWritermDagWriterThe file handle to the .dag file.protected booleanmDoneA boolean indicating whether the files have been generated or not.protected GridStartFactorymGridStartFactoryThe handle to the GridStart Factory.protected booleanmInitializeGridStartA boolean indicating whether grid start has been initialized or not.protected java.lang.StringmPoolClassSpecifies the implementing class for the pool interface.static java.lang.StringmSeparatorThe nice separator, define once, use often.private SiteStoremSiteStoreThe handle to the site catalog store.protected CondorStyleFactorymStyleFactoryHandle to the Style factory, that is used for this workflow.protected TransformationCatalogmTCHandleHandle to the Transformation Catalog.protected java.lang.StringmTempLogFileThe name of the log file in the /tmp directory-
Fields inherited from class edu.isi.pegasus.planner.code.generator.Abstract
mBag, mLogger, mPOptions, mProps, mSubmitFileDir
-
Fields inherited from interface edu.isi.pegasus.planner.code.CodeGenerator
VERSION
-
-
Constructor Summary
Constructors Constructor Description CondorGenerator()The default constructor.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected voidapplyStyle(Job job, java.io.PrintWriter writer)Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM.protected voidconcreteDagEmpty(java.lang.String filename, ADag dag)A callout method that dictates what needs to be done in case the concrete plan that is generated is empty.protected JobconstructDAGJob(java.lang.String name, java.lang.String directory, java.lang.String dagBasename)Constructs a job that plans and submits the partitioned workflow, referred to by a Partition.private java.util.MapconstructMap(java.lang.String propValue)Constructs a map with the numbers/values to be passed in the RSL handle for certain pools.private TransformationCatalogEntryconstructTCEntryFromEnvironment()Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATIONprivate TransformationCatalogEntryconstructTCEntryFromEnvProfiles(ENV env)Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATIONprivate TransformationCatalogEntrydefaultTCEntry(java.lang.String site)Returns a default TC entry to be used in case entry is not found in the transformation catalog.java.util.Collection<java.io.File>generateCode(ADag dag)Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format.voidgenerateCode(ADag dag, Job job)Generates the code (condor submit file) for a single job.protected booleangenerateLogFileSymlink(java.lang.String logFile, java.lang.String symlink)This method generates a symlink to the actual log file written in the local temp directory.java.util.Map<java.lang.String,java.lang.String>getAdditionalBraindumpEntries(ADag workflow)Returns a Map containing additional braindump entries that are specific to a Code Generator.protected java.lang.StringgetBasename(java.lang.String prefix, java.lang.String suffix)A covenience method to construct the basename.protected java.lang.StringgetCategoryDAGManKnobs(PegasusProperties properties)Write out the DAGMan knobs for each category the user mentions in the properties.protected java.lang.StringgetCondorLogInSubmitDirectory()Returns the path to the condor log file in the submit directory.protected java.lang.StringgetCondorLogInSubmitDirectory(ADag dag)Returns the path to the condor log file in the submit directory.protected java.lang.StringgetCondorLogInTmpDirectory()Returns the name of Condor log file in a tmp directory that is created if generation of symlink for condor logs is turned on.protected java.lang.StringgetDAGMANOutFilename(ADag dag)Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow.protected intgetJobPriority(Job job, int depth)Computes the priority for a job based on job type and depth in the workflowprivate java.lang.Stringgridstart(java.io.PrintWriter writer, Job job, boolean isGlobusJob)This function creates the stdio handling with and without gridstart.protected voidhandleCondorVarForJob(Job job)It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.protected voidhandleEnvVarForJob(Job sinfo)It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.protected voidhandleGlobusRSLForJob(Job sinfo)It updates/adds the the Globus RSL parameters got through the dax that are in Job object.voidinitialize(PegasusBag bag)Initializes the Code Generator implementation.protected java.io.FileinitializeDagFileWriter(java.lang.String filename, ADag workflow)Initializes the file handler to the dag file and writes the header to it.protected booleanisInteger(java.lang.String value)Returns a boolean indicating whether the value represented is an integer or not.voidpopulatePeriodicReleaseAndRemoveInJob(Job job)Populates the periodic release and remove values in the job.protected voidprintDagString(java.lang.String str)Writes a string to the dag file.voidreset()Resets the Code Generator implementation.booleanstartMonitoring()Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd.protected voidwriteDagFileTail(ADag dag)It writes the relations making up the DAG in the dag file and and closes the file handle to it.protected voidwriteDOTFile(java.lang.String filename, ADag dag)Writes out the DOT file in the submit directory.protected voidwriteJobMapFile(java.lang.String filename, ADag dag)Writes out the job map file in the submit directory.-
Methods inherited from class edu.isi.pegasus.planner.code.generator.Abstract
getDAGFilename, getDAGFilename, getFileBaseName, getWriter, writeOutBraindump, writeOutDAXReplicaStore, writeOutNotifications, writeOutStampedeEvents
-
-
-
-
Field Detail
-
DEFAULT_SUBDAG_CATEGORY_KEY
public static final java.lang.String DEFAULT_SUBDAG_CATEGORY_KEY
The default category for the sub dax jobs.- See Also:
- Constant Field Values
-
mSeparator
public static final java.lang.String mSeparator
The nice separator, define once, use often.- See Also:
- Constant Field Values
-
DEFAULT_PERIODIC_RELEASE_VALUE
public static final java.lang.String DEFAULT_PERIODIC_RELEASE_VALUE
Default value for the periodic_release for a job- See Also:
- Constant Field Values
-
DEFAULT_PERIODIC_REMOVE_VALUE
public static final java.lang.String DEFAULT_PERIODIC_REMOVE_VALUE
Default value for the periodic_remove for a job- See Also:
- Constant Field Values
-
CONDOR_DAGMAN_NAMESPACE
public static final java.lang.String CONDOR_DAGMAN_NAMESPACE
The namespace to use for condor dagman.- See Also:
- Constant Field Values
-
CONDOR_DAGMAN_LOGICAL_NAME
public static final java.lang.String CONDOR_DAGMAN_LOGICAL_NAME
The logical name with which to query the transformation catalog for the condor_dagman executable, that ends up running the mini dag as one job.- See Also:
- Constant Field Values
-
DAGMAN_PROPERTIES_PREFIX
public static final java.lang.String DAGMAN_PROPERTIES_PREFIX
The prefix for DAGMan specific properties- See Also:
- Constant Field Values
-
DEFAULT_STAGE_IN_PRIORITY_KEY
public static final int DEFAULT_STAGE_IN_PRIORITY_KEY
The default priority key associated with the stagein jobs.- See Also:
- Constant Field Values
-
DEFAULT_INTER_SITE_PRIORITY_KEY
public static final int DEFAULT_INTER_SITE_PRIORITY_KEY
The default priority key associated with the inter site transfer jobs.- See Also:
- Constant Field Values
-
DEFAULT_CREATE_DIR_PRIORITY_KEY
public static final int DEFAULT_CREATE_DIR_PRIORITY_KEY
The default priority key associated with the create dir jobs.- See Also:
- Constant Field Values
-
DEFAULT_CHMOD_PRIORITY_KEY
public static final int DEFAULT_CHMOD_PRIORITY_KEY
The default priority key associated with chmod jobs.- See Also:
- Constant Field Values
-
DEFAULT_STAGE_OUT_PRIORITY_KEY
public static final int DEFAULT_STAGE_OUT_PRIORITY_KEY
The default priority key associated with the stage out jobs.- See Also:
- Constant Field Values
-
DEFAULT_REPLICA_REG_PRIORITY_KEY
public static final int DEFAULT_REPLICA_REG_PRIORITY_KEY
The default priority key associated with the replica registration jobs.- See Also:
- Constant Field Values
-
DEFAULT_CLEANUP_PRIORITY_KEY
public static final int DEFAULT_CLEANUP_PRIORITY_KEY
The default priority key associated with the cleanup jobs.- See Also:
- Constant Field Values
-
mTCHandle
protected TransformationCatalog mTCHandle
Handle to the Transformation Catalog.
-
mSiteStore
private SiteStore mSiteStore
The handle to the site catalog store.
-
mPoolClass
protected java.lang.String mPoolClass
Specifies the implementing class for the pool interface. Contains the name of the class that implements the pool interface the user has asked at runtime.
-
mDagWriter
protected java.io.PrintWriter mDagWriter
The file handle to the .dag file. A part of the dag file is printed as we write the submit files, to insert the appropriate postscripts for handling exit codes.
-
mTempLogFile
protected java.lang.String mTempLogFile
The name of the log file in the /tmp directory
-
mDone
protected boolean mDone
A boolean indicating whether the files have been generated or not.
-
mConcreteWorkflow
protected ADag mConcreteWorkflow
The workflow for which the code has to be generated.
-
mStyleFactory
protected CondorStyleFactory mStyleFactory
Handle to the Style factory, that is used for this workflow.
-
mGridStartFactory
protected GridStartFactory mGridStartFactory
The handle to the GridStart Factory.
-
mInitializeGridStart
protected boolean mInitializeGridStart
A boolean indicating whether grid start has been initialized or not.
-
mCondorVersion
private long mCondorVersion
The long value of condor version.
-
mAssignDefaultJobPriorities
private boolean mAssignDefaultJobPriorities
Boolean indicating whether to assign job priorities or not.
-
-
Method Detail
-
initialize
public void initialize(PegasusBag bag) throws CodeGeneratorException
Initializes the Code Generator implementation. Initializes the various writers.- Specified by:
initializein interfaceCodeGenerator- Overrides:
initializein classAbstract- Parameters:
bag- the bag of initialization objects.- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
generateCode
public java.util.Collection<java.io.File> generateCode(ADag dag) throws CodeGeneratorException
Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format. Returns only the File object for the DAG file that is written out.- Parameters:
dag- the concrete workflow.- Returns:
- the Collection of
Fileobjects for the files written out. - Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
generateCode
public void generateCode(ADag dag, Job job) throws CodeGeneratorException
Generates the code (condor submit file) for a single job.- Parameters:
dag- the dag of which the job is a part of.job- theJobobject holding the information about that particular job.- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
startMonitoring
public boolean startMonitoring() throws org.griphyn.vdl.euryale.VTorInUseExceptionStarts monitoring of the workflow by invoking a workflow monitor daemon tailstatd. The tailstatd is picked up from the default path of $PEGASUS_HOME/bin/tailstatd.- Specified by:
startMonitoringin interfaceCodeGenerator- Overrides:
startMonitoringin classAbstract- Returns:
- boolean indicating whether could successfully start the monitor daemon or not.
- Throws:
org.griphyn.vdl.euryale.VTorInUseException- in case the method is called before the submit files have been generated.
-
reset
public void reset() throws CodeGeneratorExceptionResets the Code Generator implementation.- Specified by:
resetin interfaceCodeGenerator- Overrides:
resetin classAbstract- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
constructMap
private java.util.Map constructMap(java.lang.String propValue)
Constructs a map with the numbers/values to be passed in the RSL handle for certain pools. The user ends up specifying these through the properties file. The value of the property is of the form poolname1=value,poolname2=value....- Parameters:
propValue- the value of the property got from the properties file.- Returns:
- Map
-
constructDAGJob
protected Job constructDAGJob(java.lang.String name, java.lang.String directory, java.lang.String dagBasename)
Constructs a job that plans and submits the partitioned workflow, referred to by a Partition. The main job itself is a condor dagman job that submits the concrete workflow. The concrete workflow is generated by running the planner in the prescript for the job.- Parameters:
name- the name to be assigned to the job.directory- the submit directory where the submit files for the partition should reside. this is where the dag file is createddagBasename- the basename of the dag file created.- Returns:
- the constructed DAG job.
-
defaultTCEntry
private TransformationCatalogEntry defaultTCEntry(java.lang.String site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.- Parameters:
site- the site for which the default entry is required.- Returns:
- the default entry.
-
constructTCEntryFromEnvironment
private TransformationCatalogEntry constructTCEntryFromEnvironment()
Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION- Returns:
- the constructed entry else null.
-
constructTCEntryFromEnvProfiles
private TransformationCatalogEntry constructTCEntryFromEnvProfiles(ENV env)
Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION- Parameters:
env- the environment profiles.- Returns:
- the entry constructed else null if environment variables not defined.
-
getBasename
protected java.lang.String getBasename(java.lang.String prefix, java.lang.String suffix)A covenience method to construct the basename.- Parameters:
prefix- the first half of basenamesuffix- the latter half of basename- Returns:
- basename
-
initializeDagFileWriter
protected java.io.File initializeDagFileWriter(java.lang.String filename, ADag workflow) throws CodeGeneratorExceptionInitializes the file handler to the dag file and writes the header to it.- Parameters:
filename- basename of dag file to be written.dag- the workflow- Returns:
- the File object for the DAG file.
- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
getCategoryDAGManKnobs
protected java.lang.String getCategoryDAGManKnobs(PegasusProperties properties)
Write out the DAGMan knobs for each category the user mentions in the properties.- Parameters:
properties- the pegasus properties- Returns:
- the String
-
writeDOTFile
protected void writeDOTFile(java.lang.String filename, ADag dag) throws CodeGeneratorExceptionWrites out the DOT file in the submit directory.- Parameters:
filename- basename of dot file to be written .dag- theADagobject.- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
writeJobMapFile
protected void writeJobMapFile(java.lang.String filename, ADag dag) throws CodeGeneratorExceptionWrites out the job map file in the submit directory.- Parameters:
filename- basename of dot file to be written .dag- theADagobject.- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
writeDagFileTail
protected void writeDagFileTail(ADag dag) throws CodeGeneratorException
It writes the relations making up the DAG in the dag file and and closes the file handle to it.- Parameters:
dag- the executable workflow- Throws:
CodeGeneratorException
-
printDagString
protected void printDagString(java.lang.String str) throws CodeGeneratorExceptionWrites a string to the dag file. When calling this function the file handle to file is already initialized.- Parameters:
str- The String to be printed to the dag file.- Throws:
CodeGeneratorException
-
getCondorLogInTmpDirectory
protected java.lang.String getCondorLogInTmpDirectory()
Returns the name of Condor log file in a tmp directory that is created if generation of symlink for condor logs is turned on.- Returns:
- the name of the log file.
-
getCondorLogInSubmitDirectory
protected java.lang.String getCondorLogInSubmitDirectory()
Returns the path to the condor log file in the submit directory. It can be a symlink.- Parameters:
dag- the concrete workflow.- Returns:
- the path to condor log file in the submit directory.
-
getCondorLogInSubmitDirectory
protected java.lang.String getCondorLogInSubmitDirectory(ADag dag)
Returns the path to the condor log file in the submit directory. It can be a symlink.- Parameters:
dag- the concrete workflow.- Returns:
- the path to condor log file in the submit directory.
-
getAdditionalBraindumpEntries
public java.util.Map<java.lang.String,java.lang.String> getAdditionalBraindumpEntries(ADag workflow)
Returns a Map containing additional braindump entries that are specific to a Code Generator.- Specified by:
getAdditionalBraindumpEntriesin classAbstract- Parameters:
workflow- the executable workflow- Returns:
- Map containing entries for dag and condor_log
-
generateLogFileSymlink
protected boolean generateLogFileSymlink(java.lang.String logFile, java.lang.String symlink)This method generates a symlink to the actual log file written in the local temp directory. The symlink is placed in the dag directory.- Parameters:
logFile- the full path to the log file.symlink- the full path to the symlink.- Returns:
- boolean indicating if creation of symlink was successful or not
-
getDAGMANOutFilename
protected java.lang.String getDAGMANOutFilename(ADag dag)
Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow. The basename of the .out file is dependant on whether the basename prefix has been specified at runtime or not by command line options.- Parameters:
dag- the DAG containing the concrete workflow- Returns:
- the name of the dagfile.
-
concreteDagEmpty
protected void concreteDagEmpty(java.lang.String filename, ADag dag) throws CodeGeneratorExceptionA callout method that dictates what needs to be done in case the concrete plan that is generated is empty. It just logs a message saying the plan is empty.- Parameters:
filename- Filename of the dag to be written of type String.dag- the concrete dag that is empty.- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
handleCondorVarForJob
protected void handleCondorVarForJob(Job job) throws CodeGeneratorException
It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of Condor variables from various sources the following order is followed,property file, pool config file and then dax.- Parameters:
job- The Job object containing the information about the job.- Throws:
CodeGeneratorException
-
populatePeriodicReleaseAndRemoveInJob
public void populatePeriodicReleaseAndRemoveInJob(Job job)
Populates the periodic release and remove values in the job. If an integer value is specified it is used to construct the default expression, else the value specified in the profiles is used as is. The default expression for periodic_release and periodic_remove isperiodic_release = False periodic_remove = (JobStatus == 5) && ((CurrentTime - EnteredCurrentStatus) > 14400)
where releasevalue is value of condor profile periodic_release and removevalue is value of condor profile periodic_remove- Parameters:
job- the job object.
-
isInteger
protected boolean isInteger(java.lang.String value)
Returns a boolean indicating whether the value represented is an integer or not.- Parameters:
value- the String passed- Returns:
- true if an int else false
-
applyStyle
protected void applyStyle(Job job, java.io.PrintWriter writer) throws CodeGeneratorException
Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM. If no style is associated with the job, then for the job running on local site, condor style is applied. For a job running on non local sites, globus style is applied if none is associated with the job.- Parameters:
job- the job on which the style needs to be applied.writer- the PrintWriter stream to the submit file for the job.- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
handleEnvVarForJob
protected void handleEnvVarForJob(Job sinfo)
It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of environment variables from various sources the following order is followed,property file, transformation catalog, pool config file and then dax. At present values are not picked from the properties file.- Parameters:
job- The Job object containing the information about the job.
-
handleGlobusRSLForJob
protected void handleGlobusRSLForJob(Job sinfo)
It updates/adds the the Globus RSL parameters got through the dax that are in Job object. In addition inserts the additional rsl attributes that can be specified in the properties file or the pool config files in the profiles tags. In case of clashes of RSL attributes from various sources the following order is followed,property file, pool config file and then dax.- Parameters:
job- The Job object containing the information about the job.
-
getJobPriority
protected int getJobPriority(Job job, int depth)
Computes the priority for a job based on job type and depth in the workflow- Parameters:
job- the job whose priority needs to be computeddepth- the depth in the workflow- Returns:
-
gridstart
private java.lang.String gridstart(java.io.PrintWriter writer, Job job, boolean isGlobusJob) throws CodeGeneratorExceptionThis function creates the stdio handling with and without gridstart. Please note that gridstart will become the default by end 2003, and no gridstart support will be phased out.- Parameters:
writer- is an open stream for the Condor submit file.job- is the job information structure.isGlobusJob- istrue, if the job generated a lineuniverse = globus, and thus runs remotely. Set tofalse, if the job runs on the submit host in any way.- Returns:
- A possibly empty string which contains things that need to be added to the "globusrsl" clause. The return value is only of interest for isGlobusJob==true calls.
- Throws:
CodeGeneratorException- in case of any error occuring code generation.
-
-