mvapich2 block or dead lock with default BLACS topology Jing Zhang https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=465236 2013-02-22T00:32:13Z 2013-02-22T00:28:28Z

Hi, it seems to be a bug for mvapich2/1.9a2, when calling BLACS functions &#039;dgebs2d&#039; and &#039;dgebr2d&#039; with TOP=&#039; &#039;. All processes are blocked or dead-lock (not sure), will never return from &#039;dgebs2d&#039; or &#039;dgebr2d&#039;. The sample code is as follow,<br /><br /><div class="code"><span class="code-lines">&nbsp;1</span>c&nbsp; &nbsp; bcast.f<br /><span class="code-lines">&nbsp;2</span><br /><span class="code-lines">&nbsp;3</span>&nbsp; &nbsp;&nbsp; &nbsp;program test<br /><span class="code-lines">&nbsp;4</span>c------------------------------------------------------------<br /><span class="code-lines">&nbsp;5</span>&nbsp; &nbsp;&nbsp; &nbsp;include &#039;mpif.h&#039;<br /><span class="code-lines">&nbsp;6</span>&nbsp; &nbsp;&nbsp; &nbsp;integer nprocs,me<br /><span class="code-lines">&nbsp;7</span>&nbsp; &nbsp;&nbsp; &nbsp;integer nprow,npcol,ictxt,myrow,mycol<br /><span class="code-lines">&nbsp;8</span>&nbsp; &nbsp;&nbsp; &nbsp;integer n<br /><span class="code-lines">&nbsp;9</span>&nbsp; &nbsp;&nbsp; &nbsp;parameter(n=1000)<br /><span class="code-lines">10</span>&nbsp; &nbsp;&nbsp; &nbsp;double precision A(n,n)<br /><span class="code-lines">11</span><br /><span class="code-lines">12</span>&nbsp; &nbsp;&nbsp; &nbsp;call blacs_pinfo(me,nprocs)<br /><span class="code-lines">13</span>&nbsp; &nbsp;&nbsp; &nbsp;nprow=int(sqrt(float(nprocs)))<br /><span class="code-lines">14</span>&nbsp; &nbsp;&nbsp; &nbsp;npcol=int(nprocs/nprow)<br /><span class="code-lines">15</span>&nbsp; &nbsp;&nbsp; &nbsp;if(nprow*npcol.gt.nprocs)npcol=npcol-1<br /><span class="code-lines">16</span>&nbsp; &nbsp;&nbsp; &nbsp;print *,&#039;nprow=&#039;,nprow,&#039; npcol=&#039;,npcol<br /><span class="code-lines">17</span><br /><span class="code-lines">18</span>&nbsp; &nbsp;&nbsp; &nbsp;call blacs_get(-1, 0, ictxt)<br /><span class="code-lines">19</span>&nbsp; &nbsp;&nbsp; &nbsp;call blacs_gridinit(ictxt,&#039;R&#039;,nprow,npcol)<br /><span class="code-lines">20</span>&nbsp; &nbsp;&nbsp; &nbsp;call blacs_gridinfo(ictxt,nprow,npcol,myrow,mycol)<br /><span class="code-lines">21</span>&nbsp; &nbsp;&nbsp; &nbsp;print *,&#039;myrow=&#039;,myrow,&#039; mycol=&#039;,mycol<br /><span class="code-lines">22</span><br /><span class="code-lines">23</span>&nbsp; &nbsp;&nbsp; &nbsp;if(myrow.eq.0.and.mycol.eq.0)then<br /><span class="code-lines">24</span>&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; call dgebs2d(ictxt,&#039;All&#039;,&#039; &#039;,n,n,A,n)<br /><span class="code-lines">25</span>&nbsp; &nbsp;&nbsp; &nbsp;else<br /><span class="code-lines">26</span>&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; call dgebr2d(ictxt,&#039;All&#039;,&#039; &#039;,n,n,A,n,0,0)<br /><span class="code-lines">27</span>&nbsp; &nbsp;&nbsp; &nbsp;endif<br /><span class="code-lines">28</span>&nbsp; &nbsp;&nbsp; &nbsp;print *,&#039;myrow=&#039;,myrow,&#039; mycol=&#039;,mycol, &#039;pass&#039;<br /><span class="code-lines">29</span><br /><span class="code-lines">30</span>&nbsp; &nbsp;&nbsp; &nbsp;if(myrow.ge.0)call blacs_gridexit(ictxt)<br /><span class="code-lines">31</span>&nbsp; &nbsp;&nbsp; &nbsp;call blacs_exit(0)<br /><span class="code-lines">32</span>&nbsp; &nbsp;&nbsp; &nbsp;end<br /></div><br /><br />The code is compiled and executed like this,<br /><div class="code"><span class="code-lines">1</span>/opt/apps/intel13/mvapich2/1.9/bin/mpif77 -O2 -traceback&nbsp; -o bcast.x bcast.f -L/opt/apps/intel/13/composer_xe_2013.2.146/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core&nbsp; -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm<br /><span class="code-lines">2</span>ibrun -n 4 -o 0 ./bcast.x<br /></div><br /><br />If the TOP is set to like &#039;i-ring&#039; or &#039;1-tree&#039;, the code works well. Or if intel MPI is used instead of mvapich2, everything is fine. <br /><br />Is there some magic env or setting could fix this issue? Or is this a bug need to be fixed?<br /><br />Thanks,<br />Jing

Jing Zhang 2013-02-22T00:28:28Z