RE: Problem running mpi code on two nodes of SDSC gordon Mahidhar Tatineni https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=1206791 2016-04-26T21:58:45Z 2016-04-26T21:58:14Z

From your description it looks like you have lot of small files (13K+). I think you are likely hitting a meta data bottleneck and thats most likely due to just a much higher user load now. Lustre just has two metadata servers and all the meta data load is going through them. December was pretty lightly loaded and now we are running at 85-90%. Also, the load on the Lustre filesystem is higher too. On the larger files, the I/O is more bandwidth limited (meta data is fine because you only have 1023 files) and hence the lower impact. <br /><br />Given that the files are small ( 39MB ), it might be best to try this run out of the local scratch on our compute nodes. On the normal compute nodes we have close to 225GB of SSD space available local to each node. The IOPs you get should be much much higher and help on this kind of I/O. We also have some nodes with 1.5TB of SSD space if needed. Can you give us more info on the code and how it does I/O. I can set up the scripts for you to use the local storage. This might be better done via a ticket so please send an email to help@xsede.org with the details.<br /><br />Thanks,<br />Mahidhar

Mahidhar Tatineni 2016-04-26T21:58:14Z Mahidhar Tatineni https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=810923 2014-11-11T23:23:25Z 2014-11-11T23:23:25Z

Christopher,<br /><br />Can you ssh between two login nodes without a password? If not, there will be an issue. Can you send us a ticket (help@xsede.org) with your username? I can take a look and resolve any issue.<br /><br />-Mahidhar

Mahidhar Tatineni 2014-11-11T23:23:25Z Mahidhar Tatineni https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=428501 2012-12-17T16:42:36Z 2012-12-17T16:42:23Z

Mahendrakar,<br /><br />Unfortunately, we don&#039;t have a production tool for this. However, there is a topology service that was developed as part of a research project and I will send you usage info as soon as I set up the permissions for you to use it.<br /><br />-Mahidhar

Mahidhar Tatineni 2012-12-17T16:42:23Z Mahidhar Tatineni https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=419394 2012-11-27T17:01:08Z 2012-11-27T17:01:08Z

Hi Mahendrakar,<br /><br />Can you give me more info, particularly the nodes that were used? I assume you did a qsub -I to pick up the compute nodes? I see that your compute node access is set up fine.<br /><br /><br />-Mahidhar

Mahidhar Tatineni 2012-11-27T17:01:08Z