Running large grids using OpenFOAM on XSEDE Kraken Mark William Crosskey https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=668171 2014-03-07T20:24:16Z 2014-02-24T18:14:51Z

Karthik,<br /><br />I believe I&#039;ve found a way around decomposePar not running in parallel. The solution is to use redistributePar as it says in the header file:<br /><br /># Create empty processor directories (have to exist for argList)<br />mkdir processor0<br />..<br />mkdir processorN<br /># Copy undecomposed polyMesh<br />cp -r constant processor0<br /># Distribute<br />mpirun -np ddd redistributePar -parallel (for Kraken this would be aprun -n redistributePar -parallel)<br /><br />However, this leaves out a very important line on the second step. You also have to do &#034;cp -r 0 processor0&#034; or the fields will not be distributed with the mesh. Once the results are finished, you also cannot recombine them with recomposePar as that utility can also not be run in parallel. Therefore, you can create the something.openfoam file and read in each section of the grid or all of them at once from the processor directories in Fieldview/Paraview using the OpenFOAM direct reader.<br /><br />-Mark

Mark William Crosskey 2014-02-24T18:14:51Z Karthik Rudra Reddy https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=582011 2014-01-30T18:57:27Z 2013-08-23T00:44:38Z

Hi all,<br /><br />I&#039;ve been running simulations using OpenFOAM on Kraken for sometime now. Recently I tried running a grid with ~10M cells, but ran into memory issues while trying to do domain decomposition (using decomposePar). I also tried creating some simple grids using OpenFOAM&#039;s blockMesh utility to see how large a grid it can handle. The utility was able to generate grids with ~6M cells, and for larger grids, would return this error message:<br /><br />new cannot satisfy memory request.<br />This does not necessarily mean you have run out of virtual memory.<br />It could be due to a stack violation caused by e.g. bad use of pointers or an out of date shared library<br />Aborted<br /><br />This was while I was working on the login nodes (with 8GB memory). I recently submitted a ticket regarding this, and tech support suggested I use the compute nodes instead (with 16GB memory). This did take care of the memory issue. However, I&#039;m not really sure if this is a memory issue and not some other bug, since ~10M cells is not a big grid by today&#039;s standards. So I&#039;m curious to know if other people running OpenFOAM have encountered this error while working with large grids (&gt;10M cells). I&#039;d really appreciate if people who&#039;ve experienced this could comment here.<br /><br />Thanks,<br />Karthik

Karthik Rudra Reddy 2013-08-23T00:44:38Z