I/O 3x Slower Richard Barnes https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=1206826 2016-04-26T23:36:36Z 2016-04-26T23:36:36Z

I am also wondering this.

Richard Barnes 2016-04-26T23:36:36Z Richard Barnes https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=1206082 2016-04-26T04:56:16Z 2016-04-26T04:56:16Z

Everything seems to be the same, but I/O is taking 3x longer - why might that be?<br /><br /><br />I have a series of tests I ran on a system in December. The program hasn&#039;t changed since then and, as far as I know, the data hasn&#039;t either; I have sole user-level access to the data.<br /><br />The directory containing the data came from `mkdir` without any special striping permissions and contains the 13,389 files of 39MB each comprising the dataset.<br /><br />The directory is located within: /oasis/scratch/comet/$USER/temp_project<br /><br />I ran the dataset again today and it posted a much longer time-to-completion.<br /><br />Calculation times are the same to within 3%.<br /><br />I/O time has increased from 24815s (total) to 680862s (total), or 2.7x longer. The writing portion seems to be taking disproportionately longer: 2.7x longer versus 2.1x longer.<br /><br />I have a similar data set of 1023 files of 468MB each comprising a second dataset. In this case I/O is taking 1.1x longer (8272s versus 7497s).<br /><br />The only real hypothesis I have is that all my files are on the same drive (not sure what the default policy is) or a set of drives and someone else happens to be making heavy use of those.<br /><br />Is there a way to verify this?<br /><br />Could something else be the cause?

Richard Barnes 2016-04-26T04:56:16Z