Flash scratch storage - what happens if job runs out of walltime? Carl Lemmon https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=2986037 2022-05-12T21:24:03Z 2022-05-12T21:24:03Z

Hi all,<br /><br />I currently have a trial allocation on expanse and I am trying to get my software (ORCA) configured as best as possible before I run some benchmarks.<br /><br />Often I am running restartable jobs, and will run out of walltime and need to resubmit the job. I would like to use the SSD flash scratch storage to hold my running jobs, the storage that is accessed under &#034;/scratch/$USER/job/$SLURM_JOB_ID&#034;. The guide specifies this is only accessible during a job run. Obviously, after the job is done, whether it finishes successfully or fails, it goes to my next command in the slurm script which is to move all of the data back to home directory. But if it runs out of walltime while ORCA is still running, that next command to move the data back to home will not run. I was wondering what I can do to ensure that I am not wasting my SUs of that job by losing all of the data.<br /><br />Other clusters have a &#034;orphan&#034; folder where this lost data is sometimes held for a time. Do XSEDE clusters or expanse have an option like this?<br /><br />Alternatively, I have used epilog scripts before in TORQUE, and I know SLURM also has the option of epilog scripts. Will an epilog script have access to that scratch folder, and if so can I use an epilog script to copy it back so that it will always copy whether the job runs out of walltime or not?

Carl Lemmon 2022-05-12T21:24:03Z