General Discussion

Flash scratch storage - what happens if job runs out of walltime?

2022-05-12T21:24:03Z

Hi all,

I currently have a trial allocation on expanse and I am trying to get my software (ORCA) configured as best as possible before I run some benchmarks.

Often I am running restartable jobs, and will run out of walltime and need to resubmit the job. I would like to use the SSD flash scratch storage to hold my running jobs, the storage that is accessed under "/scratch/$USER/job/$SLURM_JOB_ID". The guide specifies this is only accessible during a job run. Obviously, after the job is done, whether it finishes successfully or fails, it goes to my next command in the slurm script which is to move all of the data back to home directory. But if it runs out of walltime while ORCA is still running, that next command to move the data back to home will not run. I was wondering what I can do to ensure that I am not wasting my SUs of that job by losing all of the data.

Other clusters have a "orphan" folder where this lost data is sometimes held for a time. Do XSEDE clusters or expanse have an option like this?

Alternatively, I have used epilog scripts before in TORQUE, and I know SLURM also has the option of epilog scripts. Will an epilog script have access to that scratch folder, and if so can I use an epilog script to copy it back so that it will always copy whether the job runs out of walltime or not?

Help with `srun nvidia-smi` for power measurements

2022-04-27T14:25:43Z

I'm currently writing a script to get power measurements from the NVIDIA V100 GPU on Expanse-GPU (SDSC). Out of the three methods possible (`nvidia-smi`, NVML, and CUPTI), it seems that `nvidia-smi` has the lowest barrier to entry because it comes with the NVIDIA drivers.

Currently I have the following three concerns:

1. How to deal with Slurm.
For some reason, at Expanse-GPU, after getting an interactive node on the `gpu-shared` partition (or any other Expanse-GPU partition, for that matter) , you can only run `srun nvidia-smi` but not `nvidia-smi` directly. This suggests that either Expanse has a three-tiered node hierarchy (login, batch, compute) like Summit, or for some reason the environment isn't setup properly outside of `srun`. Which one of these two scenarios is true?

2. How to spawn and kill `nvidia-smi` properly.
This is a technical aspect, but I'm not sure how to reliably spawn and kill (or start and stop) `nvidia-smi` in the Slurm job script. I am only mildly familiar with `pidof` and `kill`.

3. How to set polling interval appropriately.
This is more of a conceptual thing. In quantum physics, a measurement changes the system. I think a similar concept applies here since we're polling the GPU by using `nvidia-smi`. If the frequency is too high, we might incur a performance cost. If the frequency is too low, the data will be inaccurate and the program to be analyzed might finish entirely within the polling interval. How can I determine the appropriate polling interval?

I have also submitted ticket #21526 to the Expanse help desk.

RE: Running ORCA on Expanse

2021-03-22T20:21:52Z

HI Andrew-

In general we would need to know a bit more to assist you. What is the full command you ran with the complete filename.

However, I recommend that you submit a help ticket to help@xsede.org and the SDSC staff can help you directly to install ORCA on the system.

Nicole

Running ORCA on Expanse

2021-03-21T04:07:26Z

Hello,

I received a start-up allocation for expanse and have been trying to run geometry optimizations for a research project. I have previous experience from a class I took in my graduate program but we submitted our jobs to the Comet cluster. My teacher provided us with the a bash file template and we used a script that called on ORCA from my teacher's home directory. I tried installing ORCA on Expanse by downloading the tar.xz file from the ORCA forum and transferring the file to my home directory on the Expanse cluster. I then tried to install ORCA on Expanse by using the tar -xf command and received the following:

xz: (stdin): File format not recognized
tar: Child returned status 1
tar: Error is not recoverable: exiting now

I'm not sure if I am using the wrong command, but from my understanding Expanse runs on CentOS, so I don't know why this command wouldn't work.

Has anyone else encountered this error or know how I can install ORCA to run on Expanse?

comet's gdal missing glibc dependency?

2020-07-25T20:54:12Z

I'm trying to run gdaladdo on comet and getting a "GLIBC not found" error.
I am using the gdal module but still getting this error.

Is there a way for me to fix the error in my submission script?
I tried installing with yum but got a permissions error (not surprising).

Here is my submission script:

1#!/bin/bash
2#SBATCH --job-name ="wv_mosaic_gdal"
3#SBATCH --mem=1G
4#SBATCH --partition=large-shared
5#SBATCH --time=24:00:00
6
7module purge
8module load gdal
9
10export GDAL_VRT_ENABLE_PYTHON=YES
11export GDAL_VERT_ENABLE_PYTHON
12
13gdaladdo --config BIGTIFF_OVERIEW YES -ro CETX_overview.vrt 1

and the full error:

1gdaladdo: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by gdaladdo)
2gdaladdo: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /opt/gdal/lib/libgdal.so.20)
3gdaladdo: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /opt/gdal/lib/libgdal.so.20)
4gdaladdo: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /opt/gdal/lib/libgdal.so.20)
5gdaladdo: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /opt/gdal/lib/libgdal.so.20)

Protein Sequence XSEDE Beginners Help

2020-06-04T00:07:25Z

Hello Everyone,

I am working on a bioinformatics research project related to a bacterial protein. I am planning on utilizing XSEDE resources in my research.
Actually, I have watched some XSEDE beginners tutorial which gives me the idea about what XSEDE is capable of. But I am not sure how and where to begin. I would be grateful if you guys with the similar experience could guide me through or share your experience as a beginner.

Thank you in advance.

Subha Paudel,

module not found

2020-01-11T17:35:23Z

Hello
I upload python with the command module load python3, but when I run my
program for example pyhton3 filename.py, it gives the following error no module found
matplotlib. Similarly for other python libraries, like pandas, or tensorflow.
I tries to install with the command pip install package name , but it does not work.
Mainly in the error message it says connection problem or no module is found.

Any help is appreciated. Thanks.

Iftikhar

DOE SBIR "HPC Cybersecurity" - collaborators? feedback? comments?

2019-10-04T15:54:08Z

Dear HPC community,

I would like to ask for your help.

We are currently writing an HPC cybersecurity related proposal for the DOE SBIR (Dept. of Energy Small Business Innovation & Research) program.We are currently trying to identify partners for our project who operates an HPC center, and also for people in the field to provide feedback on our proposed approach.

We are proposing to extend our OpenPMF fine-grained access control configuration solution for HPC environments. If you are knowledgeable in HPC cybersecurity and/or DOE's HPC perspective, or are interested in partnering, then please let me know.

Thanks,
Ulrich Lang
CEO
ObjectSecurity LLC

question about setting nodes/threads with spark-submit and srun

2019-02-28T20:00:43Z

I'm really new with both spark and srun. In both spark-submit and srun, there are options to set nodes and tasks. How should I write the settings so that the spark jobs are distributed on the nodes and threads set in the sbatch script? Thanks!

Publicly accessible Swift storage?

2018-07-20T17:31:00Z

I need to do some experimentation using the Swift API and it would be convenient if there was a "publicly" (i.e. XSEDE specific) Swift storage server available that I could use. Does such a thing exist?

RE: Distributed TensorFlow on Bridges?

2018-04-07T03:38:56Z

Adrian
I've been trying to figure this out recently. Apparently I am not asking for help correctly.
Did you happen to solve your distributed tensorflow problem? If so, would you be willing to share your script?

Bob

[CFP] Pycon Tutorials, Talks, & Posters

2017-11-10T02:42:59Z

Have you dreamed of visiting Cleveland, Ohio? You should submit to PyCon's Call for proposals.

Tutorial proposals — deadline is 24 November 2017 AoE.
Talk, Poster, and Education Summit proposals — deadline is 3 January 2018 AoE.

All backgrounds, topics, and levels are encouraged to submit!
If you submit early, you can get mentorship and feedback.

More info here: https://us.pycon.org/2018/speaking/

-Jackie Kazil
Speaker Co-Chair
Python Software Foundation Board Member

OpenSeesMP on Stampede2

2017-08-02T18:49:34Z

Has anyone run OpenSeesMP or OpenSeesSP on Stampede2 or know how we can do this? I placed a copy of the .exe file in the folder from which I'm working on, and there are tcl libraries installed already. I tried to run a job with OpenSeesMP using ibrun, but I get an error saying "execvp error on file OpenSeesMP.exe Permission denied". Thanks.

RE: XSEDE user direct ssh to stampede2

2017-07-20T20:06:22Z

Hello Shaowu,

Multi-factor authentication (MFA) is now required to access TACC resources directly via ssh. Please visit the URL below for information on MFA and how to set it up.

https://portal.tacc.utexas.edu/tutorials/multifactor-authentication

Please let me know if you have additional questions.

Thanks,
Chris

Chris Hempel
Director of User Services
Texas Advanced Computing Center

XSEDE user direct ssh to stampede2

2017-07-20T02:48:40Z

Dear all,

As an XSEDE user, can I directly ssh login to stampede2.tacc.utexas.edu without going through the login.xsede.org server? When I tried it, it asked for TACC token which I do not have.

Thanks,
Shaowu Bao

Jobs running abnormally slow on Comet

2017-06-25T18:40:07Z

Recently, some jobs on Comet seem to be running a factor of 4 slower than usual (that is, they run slow from the start and throughout the entire calculation). I have cancelled such jobs and and started them over, and then they run at the expected speed. This has only occurred since I started running on 32 nodes (768 cores). What could cause the exact same code to behave like this?

I am going to start to keep track of which nodes are being used... Could it be a problem with one faulty slow node? My code runs at the pace of the slowest node.

Any advice on how I might diagnose this?

For the record, this code has run successfully on Stampede (TACC) and Pleiades (NASA) and not had this problem.

Using Persistent Communication in Fortran MPI

2017-05-31T13:34:06Z

Dear community;
I attended the summer bootcamp on 2015. We discussed OpemMP, MPI and OpenACC.
In the boot camp we solved the famous problem of laplace equation. However, I am working with a more complex and bigger problem, a CFD real code.
In this code, I need to update the ghost cells at each time step. Therefore the overhead is quite high. As a result, I have decided to use persistent communication, but my ghost cells are populated with zeroes.
The code runs, but the solution is not right. As I said before, all my ghost cells are populated with zeroes, I have tried different things but nothing has worked out.
I wanted to know if someone has used the persistent communication with the Laplace equation. If so, I will greatly appreciate if you can share the code with me to compare my syntax with a wroking version. Below I show a small part of the code.

The code, calls the MPI_Subroutine where I set the communication characteristics.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!Starting up MPI
call MPI_INIT(ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD,npes,ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD,MyRank,ierr)

!Compute the size of local block (1D Decomposition)
Jmax = JmaxGlobal
Imax = ImaxGlobal/npes
if (MyRank.lt.(ImaxGlobal - npes*Imax)) then
Imax = Imax + 1
end if
if (MyRank.ne.0.and.MyRank.ne.(npes-1)) then
Imax = Imax + 2
Else
Imax = Imax + 1
endif

! Computing neighboars
if (MyRank.eq.0) then
Left = MPI_PROC_NULL
else
Left = MyRank - 1
end if

if (MyRank.eq.(npes -1)) then
Right = MPI_PROC_NULL
else
Right = MyRank + 1
end if

! Initializing the Arrays in each processor, according to the number of local nodes
Call InitializeArrays

!Creating the channel of communication for this computation,
!Sending and receiving the u_old (Ghost cells)
Call MPI_SEND_INIT(u_old(2,: ),Jmax,MPI_DOUBLE_PRECISION,Left,tag,MPI_COMM_WORLD,req(1),ierr)
Call MPI_RECV_INIT(u_old(Imax,: ),jmax,MPI_DOUBLE_PRECISION,Right,tag,MPI_COMM_WORLD,req(2),ierr)
Call MPI_SEND_INIT(u_old(Imax-1,: ),Jmax,MPI_DOUBLE_PRECISION,Right,tag,MPI_COMM_WORLD,req(3),ierr)
Call MPI_RECV_INIT(u_old(1,: ),jmax,MPI_DOUBLE_PRECISION,Left,tag,MPI_COMM_WORLD,req(4),ierr)

End Subroutine MPI_Subroutine

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

From the main code, where the do loop is, I call the MPI_STARTALL and WaitALL in each time step.

Call MPI_STARTALL(4,req,ierr)
Call MPI_WAITALL(4,req,status,ierr)

Req is an array of dimension (4) the same status.

I am using Fortran 90... Any suggestions and comments?
Thanks before hand

'wait' - a simple question

2016-03-31T15:51:24Z

Hi,

I'm new to setting up batch files, and have a question about the 'wait' command.

The goal here is to run a lot of models with the 'cloudy' code, but last time I ran my batch file, it only calculated 72 models out of ~700. So in my new batch file (part of it inserted below), I added an increasing 'offset' (from 0 to 19), then a 'wait' to wait for the first 20 models to finish, after which the offset is reset to 0. Is that possible at all? Is there a more correct way of doing this?

Batch file has been submitted but is still in the queue so I thought I'd just ask.

Thanks in advance,
Karen

#!/bin/bash -l

#----------------------------------------------------
# SLURM job script to run SIGAME on
# TACC's Stampede system.
#
#----------------------------------------------------

#SBATCH -J SIGAME-test # Job name
#SBATCH -o sigame.%j.out # Name of stdout output file (%j expands to jobId)
#SBATCH -p normal # Queue name
#SBATCH -N 20 # Total number of nodes requested (16 cores/node)
#SBATCH -n 20 # Total number of mpi tasks requested
#SBATCH -t 02:00:00 # Run time (hh:mm:ss) - 1.5 hours
#SBATCH --mail-user=...
#SBATCH --mail-type=begin # email me when the job starts
#SBATCH --mail-type=end # email me when the job finishes

#SBATCH -A ... # <-- Allocation name to charge job against

# Launch this
ibrun -o 0 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_0 &
ibrun -o 1 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_1 &
ibrun -o 2 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_2 &
ibrun -o 3 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_3 &
ibrun -o 4 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_4 &
ibrun -o 5 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_5 &
ibrun -o 6 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_6 &
ibrun -o 7 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_7 &
ibrun -o 8 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_8 &
ibrun -o 9 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_9 &
ibrun -o 10 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_10 &
ibrun -o 11 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_11 &
ibrun -o 12 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_12 &
ibrun -o 13 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_13 &
ibrun -o 14 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_14 &
ibrun -o 15 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_15 &
ibrun -o 16 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_16 &
ibrun -o 17 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_17 &
ibrun -o 18 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_18 &
ibrun -o 19 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_19 &
wait
ibrun -o 0 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_20 &
ibrun -o 1 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_21 &
ibrun -o 2 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_22 &
ibrun -o 3 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_23 &
ibrun -o 4 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_24 &
ibrun -o 5 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_25 &
ibrun -o 6 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_26 &
ibrun -o 7 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_27 &
ibrun -o 8 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_28 &
ibrun -o 9 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_29 &
ibrun -o 10 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_30 &
ibrun -o 11 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_31 &
ibrun -o 12 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_32 &
ibrun -o 13 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_33 &
ibrun -o 14 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_34 &
ibrun -o 15 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_35 &
ibrun -o 16 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_36 &
ibrun -o 17 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_37 &
ibrun -o 18 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_38 &
ibrun -o 19 -n 1 /work/04075/kpolsen/SIGAME/cloudy/source/cloudy.exe -p GMC_39 &
wait

batch renaming files already on Ranch

2015-11-03T20:27:05Z

Hi all,

I have several directories of files already on Ranch archival storage that I need to batch-rename in order to run them efficiently through some code. Some of my files have spaces in them, and others have information in the incorrect order for processing.

A couple of examples: I would like to rename the directory (and all the files therein) 'CENTER1 20150825' to 'CENTER1_20150825', and the directory (and contents) '20150606 EALA-01' to EASTLA1_20150606'.

Can I batch rename using ssh from my terminal? Do I need to stage all the files in order to rename them and then archive them all again? We're talking ~8TB of data, so staging and re-uploading would NOT be a trivial process.

I suppose I could also just wait until I actually pull them out of Ranch into Stampede for my actual computing, and rename them then. But if there is a simple way to rename that doesn't involve re-uploading them all, it'd be nice to take care of it en masse.

I am fairly new to the unix shell, so I may need more annotation than the average XSEDE user to understand the process.

Thanks!

Non-MPI jobs on comet (ie data parallelized)

2015-05-21T02:38:53Z

Please offer advice, and possibly add to your comet:/share/apps/examples/ on how to run tasks that are parallelized by splitting data, not by using MPI or open-MPI message passing.

The examples at https://portal.xsede.org/sdsc-comet are for MPI or openMPI tasks.
Use of 'ibrun' or openmpi run methods are tricky for data-split 1 cpu tasks. This is what
I end up testing, but don't know if it is making efficient use of the node's 24 cpus, and it apparently won't work for 2+ nodes:

#! /bin/bash
## env prog=blastallc.sh prodb=prot.db protin=prot.aa datad=`pwd` sbatch srun_prog.sh
#SBATCH --job-name="blasta1"
#SBATCH --output="blasta.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24
#SBATCH -t 39:55:00

export ncpu=24
if [ "X" = "X$datad" ]; then echo "ERR:datad=what?"; exit -1; fi
if [ "X" = "X$prog" ]; then echo "ERR:prog=what?"; exit -1; fi
cd $datad

## * ibrun is wacky like aprun/ bigred in that it calls $prog ncpu times !!
ibrun --npernode 1 -v $prog

# $prog is a shell script that forks out $ncpu tasks on $ncpu data parts, as in
i=0; while [ $i -lt $ncpu ]; do { bioapp --part $i &; i=$(($i+1)); } done

Data parallelization rather than MPI is the commonest method for bioinformatics and genomics apps, as the problems are data-bound and splitting data into N chunks, running single cpu tasks on each chunk works well. Only a few of the many bio-apps are MPI-aware. For example, comet has a module with an ancient 'mpiblast' which I would not recommend because it likely hasn't been updated to NCBI's current blast code in years. The above method is what I use for NCBI blast, splitting data to ncpu parts.

- Don Gilbert