Makeflow is a workflow engine for executing large complex applications on clusters, clouds, and grids. It can be used to drive several different distributed computing systems, including Condor, SGE, and the included Work Queue system. It does not require a distributed filesystem, so you can use it to harness whatever collection of machines you have available. It is typically used for scaling up data-intensive scientific applications to hundreds or thousands of cores.
|Tags||Distributed Computing cloud Cluster GRID Workflow Scientific/Engineering Clustering/Distributed Networks|
|Operating Systems||Linux Mac OS Solaris Windows (Cygwin)|
Release Notes: Support for workers hierarchies, with a master-foremen-workers paradigm. Additionally, a worker now is able to handle more than one task at a time. Support has been added for task categories. A rule can be labeled with a category, and required computational resources (disk, memory, and CPU) can be specified per category. Makeflow then automatically communicates these requirements to the work queue or condor.
Release Notes: Work Queue now allows you to specify chunks (pieces) of an input file to be used as input for worker tasks. Makeflow now supports a -i switch which outputs pre-execution analysis of the Makeflow DAG. Work Queue and Makeflow support submitting tasks to the PBS batch submission platform. The Makeflow makeflow_log_parser now ignores comments in Makeflow logs. Work Queue supports querying workers and tasks at run-time. Makeflow Input file (dependency) renaming is supported with a new "->" operator. Work Queue Tasks are now killed using SIGKILL. Work Queue Protocol based keep-alives were added to workers.
Release Notes: In WorkQueue, memory errors that lead to a SEGFAULT, race conditions in signal handling for workers, and incorrect handling of the -C option where a worker would not use the same catalog server as work_queue_pool were corrected. Tasks now have a unique process group to properly kill all task children on abort. Escape codes such as \n and \t are now properly interpreted in Makeflow files.
Release Notes: A bug in WorkQueue that resulted in a cancelled task becoming a zombie was fixed. Various corrections were made to the manual and HTML documentation. Makeflow -I and -O options now correctly the output file list to stdout. WorkQueue now correctly removes directories during cleanup.
Release Notes: WorkQueue now has a new API for logging, improved python binding support and documentation, and various other bugfixes and minor features. Makeflow now supports garbage collection of intermediate files, lexical scoping of makeflow variables, and a new MAKEFLOW keyword for recursive makeflows.