Bridges: Processing large number of large files Mohan Sun https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=2532599 2020-08-13T20:28:57Z 2020-08-13T20:27:09Z

Hey guys,<br /><br />Do you have suggestions on best way to submitting batch job on Bridges for a project with large number of large file?<br /><br />I have a projects where I need to process around 100,000 csv.gz. An average gz file is 500 MB. I am currently processing with grep | sed | awk. So approximately for each file the processing time is 0.5 hour on a RM-shared core.<br /><br />My thinking on this is that I submit 100,000 RM-shared batch where each batch process one file. But my concern is that my later jobs will have lower priority since I have already submitted/completed many jobs.<br /><br />I cannot find such mechanism in PSC documentation but I think it&#039;s a common thing for supercomputers. Can anyone help me verify that? Also any suggestions are welcome.

Mohan Sun 2020-08-13T20:27:09Z