runmany.pl

Optimizing Time to Completion with runmany • Bill Davidsen

Home Section: www.samag.com/articles/2002/0210

If you are using a computer with multiple CPUs, lots of memory, and other resources (like bandwidth), and you need to do many small, similar tasks, it makes sense to simplify the process. However, trying to do too much at once can really bog down your system. runmany is a tool I developed to help fit the load to the resources and complete jobs faster.

#!/usr/bin/perl -w
use strict;
$|++;

use vars qw/ $opt_d $opt_v /;

my $Version = '$Id: runmany,v 1.2 2002/12/21 06:42:42 felicity Stab $';

use Getopt::Std;
&getopts("dv") || &Usage;

&Usage if $#ARGV != 1; # sanity check on args present and accounted for
my($proc,$limit) = splice @ARGV, 0, 2;

print STDERR "Cmd: \"$proc\"\nNum: $limit\n" if $opt_v || $opt_d;

my $started = 0;
my $finished = 0;
while (<>) {
		 chomp;
		 unless (fork()) {
				 my $cmd;
				 if ($proc =~ m/%s/) {
						 ($cmd = $proc) =~ s/%s/$_/g;
				 } else {
						 $cmd = "${proc} $_";
				 }		 		 
				 print STDERR "Running '$cmd'\n" if ( $opt_v || $opt_d );
				 exec $cmd;
				 # we should never get here
				 die "exec $cmd failed!: $!\n";
		 }
		 ++$started;
		 my $running = $started - $finished;
		 print STDERR "$started $finished\n" if $opt_d;
		 if ($limit <= $running) {
				 wait;
				 ++$finished;
		 }
}

# wait for last children and exit clean
until ($finished == $started) {
		 print STDERR "Final wait: $started  $finished\n" if $opt_d;
		 wait;
		 ++$finished;
}
exit 0;

sub Usage
{
		 print "\n$Version\n\n"
		 . "Usage:\n"
		 . "  runmany [options] <cmd string> <count>\n"
		 . "\n"
		 . "lines are read from stdin, and the values read are used to replace\n"
		 . "any instances of \"%s\" in the <cmd string>. Copies of the modified\n"
		 . "command are started until <count> copies are running, then a new copy\n"
		 . "is started every time a command process ends.\n"
		 . "\n"
		 . "This is useful when more than one item may be processed at a time, but\n"
		 . "processing all possible commands at once could / would overload the system.\n"
		 . "\n"
		 . "Options:\n"
		 . "  -v    Verbose, output the command and count when starting\n"
		 . "  -d    Debug. Output a bunch of data while running.\n"
		 . "\n"
		 . "Example:\n"
		 . "  ls | grep '\.o\$' | runmany \"gzip -8v %s\" 3\n"
		 . "\n"
		 . "would compress three files at a time (presumable on a system with at\n"
		 . "least that many processors.\n"
		 . "\n"
		 . "Author: Bill Davidsen <davidsen\@tmr.com>\n"
		 . "\n";

		 exit 1;
}