Parallel execution in Shell

As we know GNU parallel can execute jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. Can we do it without parallel and just by more common shell commands? We will show an example here. Personally this post is for research purpose, it not very useful :p

A parallel run can be implemented as the following two scripts:

#!/bin/bash
# parallel_run.sh by Joseph <joseph@admon.org>
# 2013-10-08
MAX_TASKS=5
tmp=$$.fifo
mkfifo $tmp
exec 5<>$tmp
rm $tmp

{
   for ((i=1; i<=${MAX_TASKS}; i++))
   do
       echo;
   done
} >&5

for id in $*
do
   read
   ( sh ./run_task.sh $id; echo >&5 ) &
done <&5
wait
exec 5>&-

run_task.sh receives one parameter each time, and run in background. run_task.sh example:

#!/bin/bash
sleep $1

5 thoughts on “Parallel execution in Shell

  1. The second script is a task example, while the first one is for parallel execution. It is much easier to use considering GNU Parallel is not installed by default on most of the common systems.

  2. IMHO, Parallel is more powerful but not easy to learn to use it. While for guys with basic scripting skill, they are able to re-use the simple script above.

Leave a comment

Your email address will not be published. Required fields are marked *