Projects / GNU Parallel

GNU Parallel

GNU parallel is a shell tool for executing jobs in parallel locally or using remote computers. A job is typically a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. If you use xargs today you will find GNU parallel very easy to use, as GNU parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. If you use ppss or pexec you will find GNU parallel will often make the command easier to read. GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU parallel as input for other programs.

Tags	Text Processing parallel Parallel processing Multicore Clustering/Distributed Networks Command Line Tools Filters System Administration
Licenses	GPLv3
Operating Systems	POSIX Linux Solaris Unix Darwin FreeBSD
Implementation	Perl

Website

Download

Release tags

All releases

Recent releases

20130522 ('Rana Plaza')

22 May 2013 01:47

Release Notes: --ctrlc is now the default if --pipe is not used, making it possible to kill remotely-started jobs simply by pressing Ctrl-C. --timeout 200% now means "kill jobs which take more than twice the time of the median runtime of a job". Jobs are now distributed round-robin when having mulitiple --sshlogin, instead of filling up one --sshlogin at a time.

20130422 ('Boston')

27 Apr 2013 07:49

Release Notes: The software can now be installed in 10 seconds by running wget -O - pi.dk/3 | sh. CPU detection was added for HURD, IRIX, SCO OpenServer, and HPUX. --ctrlc will send SIGINT to tasks running on remote computers and thus kill them. --load now uses 'ps' to find the number of running processes instead of 'uptime' and is much faster. CentOS 3.9 and FreeBSD 7 compatibility bugs were fixed. Some of these required quite extensive changes, making this release beta quality. --retries now works with --onall. The new --load computation now works on FreeBSD 7. --nice works under tcsh.

20130222 ('Chelyabinsk')

23 Feb 2013 00:48

Release Notes: --resume works with --pipe. --resume-failed will go through --joblog, redo the failed jobs, and then continue like --resume. Negative positional arguments count from the end: {-1} means the last argument, {-2} the second to last. NetBSD CPU detection. --blocksize increases exponentially if it smaller than a full record. Processing n-line records (--pipe -L n) is now much faster. --tollef is obsoleted after discussion on the mailing list.

20130122 ('Jyoti Singh Pandey')

22 Jan 2013 01:19

Release Notes: --sshdelay delays starting the next SSH by the specified number of seconds. The value can be less than 1 second.

20121222 ('End of The World')

30 Dec 2012 01:42

Release Notes: --shebang-wrap now makes it possible to parallelize scripts by putting "parallel" in the shebang line (the first #! line in scripts). --delay adds a delay after starting each job (useful to avoid raceconditions and thundering herd problems). --results now uses "/" as a separator instead of TAB, and thus creates subdirectories. OpenBSD CPU detection is now performed. niceload --sensor allows negative numbers to signify a lower number, meaning the limit is lower. niceload --prg suspends a named program which is already running. niceload multiple --pid suspends process ids which are already running.