Non-MPI jobs on comet (ie data parallelized)

2015-05-21T02:38:53Z

Please offer advice, and possibly add to your comet:/share/apps/examples/ on how to run tasks that are parallelized by splitting data, not by using MPI or open-MPI message passing.

The examples at https://portal.xsede.org/sdsc-comet are for MPI or openMPI tasks.
Use of 'ibrun' or openmpi run methods are tricky for data-split 1 cpu tasks. This is what
I end up testing, but don't know if it is making efficient use of the node's 24 cpus, and it apparently won't work for 2+ nodes:

#! /bin/bash
## env prog=blastallc.sh prodb=prot.db protin=prot.aa datad=`pwd` sbatch srun_prog.sh
#SBATCH --job-name="blasta1"
#SBATCH --output="blasta.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24
#SBATCH -t 39:55:00

export ncpu=24
if [ "X" = "X$datad" ]; then echo "ERR:datad=what?"; exit -1; fi
if [ "X" = "X$prog" ]; then echo "ERR:prog=what?"; exit -1; fi
cd $datad

## * ibrun is wacky like aprun/ bigred in that it calls $prog ncpu times !!
ibrun --npernode 1 -v $prog

# $prog is a shell script that forks out $ncpu tasks on $ncpu data parts, as in
i=0; while [ $i -lt $ncpu ]; do { bioapp --part $i &; i=$(($i+1)); } done

Data parallelization rather than MPI is the commonest method for bioinformatics and genomics apps, as the problems are data-bound and splitting data into N chunks, running single cpu tasks on each chunk works well. Only a few of the many bio-apps are MPI-aware. For example, comet has a module with an ancient 'mpiblast' which I would not recommend because it likely hasn't been updated to NCBI's current blast code in years. The above method is what I use for NCBI blast, splitting data to ncpu parts.

- Don Gilbert