Package edu.isi.pegasus.planner.dax
Class ADAG
- java.lang.Object
-
- edu.isi.pegasus.planner.dax.ADAG
-
public class ADAG extends java.lang.ObjectThis class provides the Java API to create DAX files. The DAX XML SCHEMA is available at http://pegasus.isi.edu/schema/dax-3.3.xsd and documentation available at http://pegasus.isi.edu/wms/docs/schemas/dax-3.3/dax-3.3.html The DAX consists of 6 parts the first 4 are optional and the last is optional.
- file:Used as "In DAX" Replica Catalog (Optional)
- executable: Used as "In DAX" Transformation Catalog (Optional)
- transformation: Used to describe compound executables. i.e. Executable depending on other executables (Optional)
- job|dax|dag: Used to describe a single job or sub dax or sub dax. Atleast 1 required.
- child: The dependency section to describe dependencies between job|dax|dag elements. (Optional)

To generate an example DIAMOND DAX run the ADAG Class as shown below java ADAG filename NOTE: This is an illustrative example only. Please see examples directory for a working example Shown below are some of the steps in creating a DIAMOND DAX.
- Create a new
ADAGobject
ADAG dax = new ADAG("test"); - Add notifications to the
workflow
j3.addNotification(WHEN.start,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");
j3.addNotification(WHEN.at_end,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com"); - Create
a
Fileobject
You only need to add entries to this section if you want to use an "IN-DAX" Replica Catalog"
File fa = new File("f.a"); - Add
MetaDataentry to the file objects
fa.addMetaData("string", "foo", "bar");
fa.addMetaData("int", "num", "1");
- Add
Profileentry to the file objects
fa.addProfile("env", "FOO", "/usr/bar");
fa.addProfile("globus", "walltime", "40"); - Add
PFNto the File object
fa.addPhysicalFile("file:///scratch/f.a", "local"); - Add the File object to the Replica Catalog
section of the DAX
dax.addFile(fa); - Create an
Executableobject
You only need to add entries to this section if you want to use an "IN-DAX" Replica Catalog"
Executable preprocess = new Executable("pegasus", "preproces", "1.0");
- Set the
Executable.ARCHandExecutable.OSfor the executable. Default is x86 and LINUX
preprocess.setArchitecture(Executable.ARCH.x86).setOS(Executable.OS.LINUX); - Set the executable as available to be staged. Default is
installed executable
preprocess.unsetInstalled(); - Add the physical location
PFNof the executable. In case of stageable executables the path should be a url
preprocess.addPhysicalFile(new PFN("file:///opt/pegasus/default/bin/keg")); - Add
ProfileandMetaDataobjects to the executable
preprocess.addProfile(Profile.NAMESPACE.globus, "walltime", "120");
preprocess.addMetaData("string", "project", "pegasus"); - Add the
Executableobject to theADAGobject
dax.addExecutable(preprocess);
- Set the
- Create a
Transformationobject : compound Executable (Executable depending on other executable and files)
Transformation diamond = new Transformation("pegasus", "diamond", "1.0");
- Add the sub executable for this
transformation
diamond.uses(preprocess).uses(findrange).uses(analyze);
- Add the sub files(e.g config files) for this
transformation
diamond.uses(new File("config", File.LINK.INPUT));
- Finally Add the Transformation to
the
ADAGobject
dax.addTransformation(diamond);
- Add the sub executable for this
transformation
- Create a
Jobobject
Job j1 = new Job("j1", "pegasus", "preprocess", "1.0", "j1");
- Add Arguments to the job object
j1.addArgument("-a","preprocess")
j1.addArgument("-T","60").addArgument("-i",fa);
j1.addArgument("-o").addArgument(fb1).addArgument(fb2); - Add the Files that are used by this job
j1.uses(fa, File.LINK.INPUT);
j1.uses(fb1, File.LINK.OUTPUT);
j1.uses(new File("f.b2"), File.LINK.OUTPUT); - Add the Notifications
to this job
j3.addNotification(WHEN.start,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");
j3.addNotification(WHEN.at_end,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com"); - Add
Profiles to the job
j1.addProfile(Profile.NAMESPACE.dagman, "pre", "20"); - Add the Job object to
ADAG
dax.addJob(j1);
- Add Arguments to the job object
- Add a
DAGobject
DAG j2 = new DAG("j2", "findrange.dag", "j2");
j2.uses(new File("f.b1"), File.LINK.INPUT);
j2.uses(new File("f.c1"), File.LINK.OUTPUT);
j2.addProfile(Profile.NAMESPACE.dagman, "pre", "20");
j2.addProfile("condor", "universe", "vanilla");
dax.addDAG(j2); - Add a
DAXjob object.
DAX j3 = new DAX("j3", "findrange.dax", "j3");
j3.addArgument("--site").addArgument("local");
j3.uses(new File("f.b2"), File.LINK.INPUT);
j3.uses(new File("f.c2"), File.LINK.OUTPUT);
j3.addProfile("ENV", "HAHA", "YADAYADAYADA");
dax.addDAX(j3); - Add the Job dependencies
Dependencies can be added by specifiying the job id's like so
dax.addDependency("j1", "j2", "1-2").addDependency("j1", "j3", "1-3");
or by specifying the job|dax|dag objects directly as below
dax.addDependency(j1,j3); - Finally write the dax to a
file
dax.writeToFile("diamond.dax");
- Version:
- $Revision$
- Author:
- Gaurang Mehta gmehta at isi dot edu
-
-
Field Summary
Fields Modifier and Type Field Description private intmCountThe Count of the number of dax objects : Nprivate java.util.Map<java.lang.String,java.util.Set<Edge>>mDependenciesMap of Dependencies between Job,DAX,DAG objects.private java.util.Set<Executable>mExecutablesThe list of Executable objectsprivate java.util.List<File>mFilesThe list of edu.isi.pegasus.planner.dax.File objectsprivate intmIndexThe Index of the dax object.private java.util.List<Invoke>mInvokesList of Notification objectsprivate java.util.Map<java.lang.String,AbstractJob>mJobsThe List of Job,DAX and DAG objectsprivate java.util.List<DAG>mLDAGsprivate java.util.List<DAX>mLDAXsprivate java.util.List<Job>mLJobsprivate LogManagermLoggerprivate java.lang.StringmNameThe Name / Label of the DAXprivate java.util.Set<Transformation>mTransformationsThe List of Transformation objectsprivate XMLWritermWriterHandle the XML writerstatic java.lang.StringSCHEMA_LOCATIONThe "not-so-official" location URL of the DAX schema definition.static java.lang.StringSCHEMA_NAMESPACEThe "official" namespace URI of the site catalog schema.static java.lang.StringSCHEMA_NAMESPACE_XSIXSI SCHEMA NAMESPACEstatic java.lang.StringSCHEMA_VERSIONThe version to report.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description private ADAGaddAbstractJob(AbstractJob ajob)Add AbstractJob to the DAXprivate ADAGaddAbstractJobs(java.util.List<AbstractJob> ajobs)Add AbstractJobs to the DAXADAGaddDAG(DAG dag)Add a DAG job to the DAXADAGaddDAGs(java.util.List<DAG> dags)Add multiple DAG jobs to the DAXADAGaddDAX(DAX dax)Add a DAX job to the DAXADAGaddDAXs(java.util.List<DAX> daxs)Add multiple DAX jobs to the DAXADAGaddDependency(AbstractJob parent, AbstractJob child)Add a parent child dependency between two jobs,dax,dagADAGaddDependency(AbstractJob parent, AbstractJob child, java.lang.String label)Add a parent child dependency with a dependency labelADAGaddDependency(java.lang.String parent, java.lang.String child)Add a parent child dependency between two jobs,dax,dagADAGaddDependency(java.lang.String parent, java.lang.String child, java.lang.String label)Add a parent child dependency with a dependency labelADAGaddExecutable(Executable executable)Add Executable to the DAXADAGaddExecutables(java.util.List<Executable> executables)Add Multiple Executable objects to the DAXADAGaddFile(File file)Add a RC File object to the top of the DAX.ADAGaddFiles(java.util.List<File> files)Add Files to the RC Section on top of the DAXADAGaddInvoke(Invoke invoke)Add a Notification for this WorkflowADAGaddInvoke(Invoke.WHEN when, java.lang.String what)Add a Notification for this WorkflowADAGaddInvokes(java.util.List<Invoke> invokes)Add a List of Notifications for this WorkflowADAGaddJob(Job job)Add Job to the DAXADAGaddJobs(java.util.List<Job> jobs)Add multiple Jobs to the DAXADAGaddNotification(Invoke invoke)Add a Notification for this WorkflowADAGaddNotification(Invoke.WHEN when, java.lang.String what)Add a Notification for this WorkflowADAGaddNotifications(java.util.List<Invoke> invokes)Add a List of Notifications for this WorkflowADAGaddTransformation(Transformation transformation)Add Transformation to the DAXADAGaddTransformations(java.util.List<Transformation> transformations)Add Multiple Transformation to the DAXprivate booleancontainsAbstractJob(AbstractJob ajob)Check if an abstractjob exists in the DAXprivate booleancontainsAbstractJobId(java.lang.String ajobid)Check if a jobid exists in the DAXbooleancontainsDAG(DAG dag)Check if a DAG job exists in the DAXbooleancontainsDAGId(java.lang.String dagid)Check if a DAG job id exists in the DAXbooleancontainsDAX(DAX dax)Check if a DAX job exists in the DAXbooleancontainsDAXId(java.lang.String daxid)Check if a DAX job id exists in the DAXbooleancontainsExecutable(Executable executable)Checks if a given executable exists in the DAX based Transformation CatalogbooleancontainsJob(Job job)Check if a job exists in the DAXbooleancontainsJobId(java.lang.String jobid)Check if a jobid exists in the DAXbooleancontainsTransformation(Transformation transformation)Checks if a given Transformation exists in the DAX based Transformation Catalogprivate static ADAGDiamond()private AbstractJobgetAbstractJob(java.lang.String ajobid)Returns an abstract Job with id ajobid if present otherwise null.intgetCount()Returns the total count of the dax collection.DAGgetDAG(java.lang.String dagid)Returns a DAG object with id dagid if present otherwise null.java.util.List<DAG>getDAGs()Get a list of all the DAG jobs.DAXgetDAX(java.lang.String daxid)Returns a DAX object with id daxid if present otherwise null.java.util.List<DAX>getDAXs()Get a list of all the DAX jobs.java.util.Set<Edge>getEdges()Returns a Set of all the Edge objects for the DAX.java.util.Set<Edge>getEdges(java.lang.String child)Returns a list of Edge objects for a child job/dax/dag id.java.util.Set<Executable>getExecutables()Returns a set of Executable Objects stored as part of the inDAX Transformation Catalog;java.util.List<File>getFiles()Returns a list of File objects defined as the inDax Replica CatalogintgetIndex()java.util.List<Invoke>getInvoke()Returns a list of Invoke objects associated with the workflowJobgetJob(java.lang.String jobid)Returns a Job object with id jobid if present otherwise null.java.util.List<Job>getJobs()Get a list of all the DAG jobs.java.lang.StringgetName()Return the name/label of the daxjava.util.List<Invoke>getNotification()Returns a list of Invoke objects associated with the workflow.java.util.Set<Transformation>getTransformations()Returns a set of Transformation Objects (complex executables) stored in the DAX based Transformation Catalogstatic voidmain(java.lang.String[] args)Create an example DIAMOND DAXvoidtoXML(XMLWriter writer)Generates a DAX representation.voidwriteToFile(java.lang.String daxfile)Generate a DAX File out of this object;voidwriteToSTDOUT()Generate a DAX representation on STDOUT.voidwriteToWriter(java.io.Writer writer, boolean close)Generate a DAX representation and pipe it into the Writer
-
-
-
Field Detail
-
SCHEMA_NAMESPACE
public static final java.lang.String SCHEMA_NAMESPACE
The "official" namespace URI of the site catalog schema.- See Also:
- Constant Field Values
-
SCHEMA_NAMESPACE_XSI
public static final java.lang.String SCHEMA_NAMESPACE_XSI
XSI SCHEMA NAMESPACE- See Also:
- Constant Field Values
-
SCHEMA_LOCATION
public static final java.lang.String SCHEMA_LOCATION
The "not-so-official" location URL of the DAX schema definition.- See Also:
- Constant Field Values
-
SCHEMA_VERSION
public static final java.lang.String SCHEMA_VERSION
The version to report.- See Also:
- Constant Field Values
-
mName
private java.lang.String mName
The Name / Label of the DAX
-
mIndex
private int mIndex
The Index of the dax object. I out of N
-
mCount
private int mCount
The Count of the number of dax objects : N
-
mJobs
private java.util.Map<java.lang.String,AbstractJob> mJobs
The List of Job,DAX and DAG objects- See Also:
DAG,DAX,Job,AbstractJob
-
mLJobs
private java.util.List<Job> mLJobs
-
mLDAGs
private java.util.List<DAG> mLDAGs
-
mLDAXs
private java.util.List<DAX> mLDAXs
-
mTransformations
private java.util.Set<Transformation> mTransformations
The List of Transformation objects- See Also:
Transformation
-
mExecutables
private java.util.Set<Executable> mExecutables
The list of Executable objects- See Also:
Executable
-
mFiles
private java.util.List<File> mFiles
The list of edu.isi.pegasus.planner.dax.File objects- See Also:
File
-
mDependencies
private java.util.Map<java.lang.String,java.util.Set<Edge>> mDependencies
Map of Dependencies between Job,DAX,DAG objects. Map key is a string that holds the child element reference, the value is a List of Parent objects- See Also:
Parent
-
mInvokes
private java.util.List<Invoke> mInvokes
List of Notification objects
-
mWriter
private XMLWriter mWriter
Handle the XML writer
-
mLogger
private LogManager mLogger
-
-
Constructor Detail
-
ADAG
public ADAG(java.lang.String name)
The Simple constructor for the DAX object- Parameters:
name- DAX LABEL
-
ADAG
public ADAG(java.lang.String name, int index, int count)DAX Constructor- Parameters:
name- DAX Labelindex- Index of DAX out of N DAX'scount- Number of DAXS in a group
-
-
Method Detail
-
getName
public java.lang.String getName()
Return the name/label of the dax- Returns:
-
getIndex
public int getIndex()
-
getCount
public int getCount()
Returns the total count of the dax collection. (legacy)- Returns:
- int
-
addInvoke
public ADAG addInvoke(Invoke.WHEN when, java.lang.String what)
Add a Notification for this Workflow- Parameters:
when-what-- Returns:
- ADAG
-
addNotification
public ADAG addNotification(Invoke.WHEN when, java.lang.String what)
Add a Notification for this Workflow- Parameters:
when-what-- Returns:
- ADAG
-
addInvoke
public ADAG addInvoke(Invoke invoke)
Add a Notification for this Workflow- Parameters:
invoke-- Returns:
- ADAG
-
addNotification
public ADAG addNotification(Invoke invoke)
Add a Notification for this Workflow- Parameters:
invoke-- Returns:
- ADAG
-
addInvokes
public ADAG addInvokes(java.util.List<Invoke> invokes)
Add a List of Notifications for this Workflow- Parameters:
invokes-- Returns:
- ADAG
-
addNotifications
public ADAG addNotifications(java.util.List<Invoke> invokes)
Add a List of Notifications for this Workflow- Parameters:
invokes-- Returns:
- ADAG
-
getInvoke
public java.util.List<Invoke> getInvoke()
Returns a list of Invoke objects associated with the workflow- Returns:
-
getNotification
public java.util.List<Invoke> getNotification()
Returns a list of Invoke objects associated with the workflow. Same as getInvoke()- Returns:
-
addFile
public ADAG addFile(File file)
Add a RC File object to the top of the DAX.- Parameters:
file- File object to be added to the RC section- Returns:
- ADAG
- See Also:
File
-
addFiles
public ADAG addFiles(java.util.List<File> files)
Add Files to the RC Section on top of the DAX- Parameters:
files- ListList of file objects to be added to the RC Section - Returns:
- ADAG
- See Also:
File
-
getFiles
public java.util.List<File> getFiles()
Returns a list of File objects defined as the inDax Replica Catalog- Returns:
-
addExecutable
public ADAG addExecutable(Executable executable)
Add Executable to the DAX- Parameters:
executable- Executable to be added- Returns:
- ADAG
- See Also:
Executable
-
addExecutables
public ADAG addExecutables(java.util.List<Executable> executables)
Add Multiple Executable objects to the DAX- Parameters:
executables- List of Executable objects to be added- Returns:
- ADAG
- See Also:
Executable
-
getExecutables
public java.util.Set<Executable> getExecutables()
Returns a set of Executable Objects stored as part of the inDAX Transformation Catalog;- Returns:
-
containsExecutable
public boolean containsExecutable(Executable executable)
Checks if a given executable exists in the DAX based Transformation Catalog- Parameters:
executable-- Returns:
- boolean
-
addTransformation
public ADAG addTransformation(Transformation transformation)
Add Transformation to the DAX- Parameters:
transformation- Transformation object to be added- Returns:
- ADAG
- See Also:
Transformation
-
addTransformations
public ADAG addTransformations(java.util.List<Transformation> transformations)
Add Multiple Transformation to the DAX- Parameters:
transformations- List of Transformation objects- Returns:
- ADAG
- See Also:
Transformation
-
containsTransformation
public boolean containsTransformation(Transformation transformation)
Checks if a given Transformation exists in the DAX based Transformation Catalog- Parameters:
transformation- Transformation- Returns:
- boolean
-
getTransformations
public java.util.Set<Transformation> getTransformations()
Returns a set of Transformation Objects (complex executables) stored in the DAX based Transformation Catalog- Returns:
-
addAbstractJob
private ADAG addAbstractJob(AbstractJob ajob)
Add AbstractJob to the DAX- Parameters:
ajob- AbstractJob- Returns:
- ADAG
- See Also:
Job,DAG,DAX,AbstractJob
-
addAbstractJobs
private ADAG addAbstractJobs(java.util.List<AbstractJob> ajobs)
Add AbstractJobs to the DAX- Parameters:
ajobs- AbstractJob- Returns:
- ADAG
- See Also:
Job,DAG,DAX,AbstractJob
-
getAbstractJob
private AbstractJob getAbstractJob(java.lang.String ajobid)
Returns an abstract Job with id ajobid if present otherwise null.- Parameters:
ajobid-- Returns:
-
containsAbstractJob
private boolean containsAbstractJob(AbstractJob ajob)
Check if an abstractjob exists in the DAX- Parameters:
ajob-- Returns:
-
containsAbstractJobId
private boolean containsAbstractJobId(java.lang.String ajobid)
Check if a jobid exists in the DAX- Parameters:
ajobid-- Returns:
-
addJob
public ADAG addJob(Job job)
Add Job to the DAX- Parameters:
job-- Returns:
- ADAG
- See Also:
Job,AbstractJob
-
addJobs
public ADAG addJobs(java.util.List<Job> jobs)
Add multiple Jobs to the DAX- Parameters:
jobs-- Returns:
- ADAG
- See Also:
Job,AbstractJob
-
containsJob
public boolean containsJob(Job job)
Check if a job exists in the DAX- Parameters:
job-- Returns:
-
containsJobId
public boolean containsJobId(java.lang.String jobid)
Check if a jobid exists in the DAX- Parameters:
jobid-- Returns:
-
getJob
public Job getJob(java.lang.String jobid)
Returns a Job object with id jobid if present otherwise null.- Parameters:
jobid-- Returns:
-
getJobs
public java.util.List<Job> getJobs()
Get a list of all the DAG jobs.- Returns:
-
getDAXs
public java.util.List<DAX> getDAXs()
Get a list of all the DAX jobs.- Returns:
-
getDAX
public DAX getDAX(java.lang.String daxid)
Returns a DAX object with id daxid if present otherwise null.- Parameters:
daxid-- Returns:
-
getDAGs
public java.util.List<DAG> getDAGs()
Get a list of all the DAG jobs.- Returns:
-
getDAG
public DAG getDAG(java.lang.String dagid)
Returns a DAG object with id dagid if present otherwise null.- Parameters:
dagid-- Returns:
-
addDAG
public ADAG addDAG(DAG dag)
Add a DAG job to the DAX- Parameters:
dag- the DAG to be added- Returns:
- ADAG
- See Also:
DAG,AbstractJob
-
addDAGs
public ADAG addDAGs(java.util.List<DAG> dags)
Add multiple DAG jobs to the DAX- Parameters:
dags- List of DAG jobs to be added- Returns:
- ADAG
- See Also:
DAG,AbstractJob
-
containsDAG
public boolean containsDAG(DAG dag)
Check if a DAG job exists in the DAX- Parameters:
dag-- Returns:
-
containsDAGId
public boolean containsDAGId(java.lang.String dagid)
Check if a DAG job id exists in the DAX- Parameters:
dagid-- Returns:
-
addDAX
public ADAG addDAX(DAX dax)
Add a DAX job to the DAX- Parameters:
dax- DAX to be added- Returns:
- ADAG
- See Also:
DAX,AbstractJob
-
addDAXs
public ADAG addDAXs(java.util.List<DAX> daxs)
Add multiple DAX jobs to the DAX- Parameters:
daxs- LIST of DAX jobs to be added- Returns:
- ADAG
- See Also:
DAX,AbstractJob
-
containsDAX
public boolean containsDAX(DAX dax)
Check if a DAX job exists in the DAX- Parameters:
dax-- Returns:
-
containsDAXId
public boolean containsDAXId(java.lang.String daxid)
Check if a DAX job id exists in the DAX- Parameters:
daxid-- Returns:
-
addDependency
public ADAG addDependency(java.lang.String parent, java.lang.String child)
Add a parent child dependency between two jobs,dax,dag- Parameters:
parent- String job,dax,dag idchild- String job,dax,dag,id- Returns:
- ADAG
-
addDependency
public ADAG addDependency(AbstractJob parent, AbstractJob child)
Add a parent child dependency between two jobs,dax,dag- Parameters:
parent- Job|DAX|DAG objectchild- Job|DAX|DAG object- Returns:
-
addDependency
public ADAG addDependency(java.lang.String parent, java.lang.String child, java.lang.String label)
Add a parent child dependency with a dependency label- Parameters:
parent- String job,dax,dag idchild- String job,dax,dag idlabel- String dependency label- Returns:
- ADAG
-
getEdges
public java.util.Set<Edge> getEdges(java.lang.String child)
Returns a list of Edge objects for a child job/dax/dag id. Returns an empty set if the child does not have any parents Returns null if the child is not a valid job/dax/dag id- Parameters:
child-- Returns:
-
getEdges
public java.util.Set<Edge> getEdges()
Returns a Set of all the Edge objects for the DAX. Returns empty if no dependencies.- Parameters:
child-- Returns:
-
addDependency
public ADAG addDependency(AbstractJob parent, AbstractJob child, java.lang.String label)
Add a parent child dependency with a dependency label- Parameters:
parent- Job|DAX|DAG objectchild- Job|DAX|DAG objectlabel- String label for annotation- Returns:
- ADAG
-
writeToFile
public void writeToFile(java.lang.String daxfile)
Generate a DAX File out of this object;- Parameters:
daxfile- The file to write the DAX to
-
writeToSTDOUT
public void writeToSTDOUT()
Generate a DAX representation on STDOUT.
-
writeToWriter
public void writeToWriter(java.io.Writer writer, boolean close)Generate a DAX representation and pipe it into the Writer- Parameters:
writer- A Writer objectclose- Whether writer should be closed on return.
-
toXML
public void toXML(XMLWriter writer)
Generates a DAX representation.- Parameters:
writer-
-
main
public static void main(java.lang.String[] args)
Create an example DIAMOND DAX- Parameters:
args-
-
Diamond
private static ADAG Diamond()
-
-