Help with `srun nvidia-smi` for power measurements Wileam Yonatan Phan https://conferences.xsede.org/c/message_boards/find_message?p_l_id=&messageId=2975453 2022-04-27T14:25:43Z 2022-04-27T14:25:43Z

I&#039;m currently writing a script to get power measurements from the NVIDIA V100 GPU on Expanse-GPU (SDSC). Out of the three methods possible (`nvidia-smi`, NVML, and CUPTI), it seems that `nvidia-smi` has the lowest barrier to entry because it comes with the NVIDIA drivers.<br /><br />Currently I have the following three concerns:<br /><br /> 1. How to deal with Slurm.<br /> For some reason, at Expanse-GPU, after getting an interactive node on the `gpu-shared` partition (or any other Expanse-GPU partition, for that matter) , you can only run `srun nvidia-smi` but not `nvidia-smi` directly. This suggests that either Expanse has a three-tiered node hierarchy (login, batch, compute) like Summit, or for some reason the environment isn&#039;t setup properly outside of `srun`. Which one of these two scenarios is true?<br /><br />2. How to spawn and kill `nvidia-smi` properly.<br /> This is a technical aspect, but I&#039;m not sure how to reliably spawn and kill (or start and stop) `nvidia-smi` in the Slurm job script. I am only mildly familiar with `pidof` and `kill`.<br /><br />3. How to set polling interval appropriately.<br /> This is more of a conceptual thing. In quantum physics, a measurement changes the system. I think a similar concept applies here since we&#039;re polling the GPU by using `nvidia-smi`. If the frequency is too high, we might incur a performance cost. If the frequency is too low, the data will be inaccurate and the program to be analyzed might finish entirely within the polling interval. How can I determine the appropriate polling interval?<br /><br />I have also submitted ticket #21526 to the Expanse help desk.

Wileam Yonatan Phan 2022-04-27T14:25:43Z