I run a custom compiled version of MPI for my OpenFOAM simulations and sometimes, I forget the defaults. In this case, today, instead of grabbing all the cores on one of my 24 core simulation machines, I wanted to run 12 x 2-core jobs. Pretty easy I thought but NO!
I had forgotten that the default for running different MPI jobs on the one machine is that they all grab the same first cores and stomp on each other. See the screen grab below that I was fortunate to check before launching all my jobs.
Fortunately the fix is simple and I just had to remember my mpirun commands and not be lazy. Normally my control scripts execute OpenFOAM jobs with:
runParallel $(getApplication)
I just had to swap this out instead with:
mpirun -np 2 --cpu-set 2-3 $(getApplication) -parallel > log.$(getApplication)
The key is the –cpu-set option that forces the MPI application to use logical cores two and three. Remember the core count starts from zero (0).
From there all was good.
Finally, thanks for reading this article and please leave a comment below. If you are interested in being updated when similar items are posted then either subscribe via RSS or sign up to my mailing list below.
