GNU parallel

<h2 id="usage">Usage</h2>

<p>The most common usage is to replace the shell loop, for example
</p>
while read x; do 
    do_something "$x"
done < list

<p>to the form of
</p>
cat list | parallel do_something

<p>where the file list contains arguments for do_something and where process_output may be empty.
</p><p>Scripts using parallel are often easier to read than scripts using <a href="/facts/Pexec/YekmD3hV">pexec</a>.
</p><p>The program parallel features also
</p>
<ul><li>grouping of <a href="/facts/Standard_output/r92M7Wd4">standard output</a> and <a href="/facts/Standard_error_stream/r92M7Wd4">standard error</a> so the output of the parallel running jobs do not run together;</li>
<li>retaining the order of output to remain the same order as input;</li>
<li>dealing nicely with filenames containing special characters such as space, single quote, double quote, ampersand, and UTF-8 encoded characters;</li></ul>
<p>By default, parallel runs as many jobs in parallel as there are <a href="/facts/CPU/Jxp1bqMV">CPU cores</a>.
</p>
<h2 id="examples">Examples</h2>
find . -name "*.foo" | parallel grep bar

<p>The above is the parallel equivalent to:
</p>
find . -name "*.foo" -exec grep bar {} +

<p>This searches in all files in the current <a href="/facts/Directory_(file_systems)/VLtLcEXY">directory</a> and its subdirectories whose name end in .foo for occurrences of the <a href="/facts/String_(computer_science)/4ChJu5WS">string</a> bar. The parallel command will work as expected unless a file name contains a <a href="/facts/Newline/ffeJz4Nn">newline</a>. In order to avoid this limitation one may use:
</p>
find . -name "*.foo" -print0 | parallel -0 grep bar

<p>The above command uses the <a href="/facts/Null_character/IMYNHNxR">null character</a> to delimit file names.
</p>
find . -name "*.foo" | parallel -X mv {} /tmp/trash

<p>The above command expands {} with as many arguments as the command line length permits, distributing them evenly among parallel jobs if required. This can lower process overhead for short-lived commands that take less time to finish than they do to launch.
</p>
find . -maxdepth 1 -type f -name "*.ogg" | parallel -X -r cp -v -p {} /home/media

<p>The command above does the same as:
</p>
cp -v -p *.ogg /home/media

<p>However, the former command which uses find/parallel/cp is more resource efficient and will not halt with an error if the expansion of *.ogg is too large for the shell.
</p>

Wikimedia Commons has media related to GNU parallel.

<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Xargs/nXhrdwXI">xargs</a></li>
<li><a href="/facts/Pexec/YekmD3hV">pexec</a></li>
<li><a href="/facts/GNU_Queue/dvfftmf2">GNU Queue</a></li></ul>

<h2 id="external-links">External links</h2>
<ul><li><a href="https://www.gnu.org/software/parallel/">Official website</a> </li>
<li><a href="https://savannah.gnu.org/projects/parallel/">Project page and manual page of GNU parallel</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>"GNU Parallel". GNU.org. <a href="https://www.gnu.org/software/parallel/" target="_blank">https://www.gnu.org/software/parallel/</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
</ol>

GNU parallel open-in-new

GNU parallel