mpi issues  [SOLVED]

Total energy, geometry optimization, DFT+U, spin....

Moderator: bguster

Locked
baisxun
Posts: 6
Joined: Fri Mar 30, 2018 5:10 am

mpi issues

Post by baisxun » Mon May 07, 2018 3:10 am

Hi,

I have a problem with my mpi paramaters, in the crystal system I'm calculating, there are 167 bands, and when I run on a cluster, a message as follows constantly comes up:

"You should try to get npband*bandpp=167
For information matrix size is 177655"

For each step this message comes up several times before actually doing any calculation and as a result mpi is much slower than non-mpi runs. The problem is, I can't come up with a combination of npband and bandpp that gives 167, it's just impossible.

Has anyone else ever encountered such an issue? How did you deal with it?

Any comments would be most appreciated.

ebousquet
Posts: 469
Joined: Tue Apr 19, 2011 11:13 am
Location: University of Liege, Belgium

Re: mpi issues

Post by ebousquet » Mon May 07, 2018 12:58 pm

Dear baisxun,
You can increase nband to >167 that can fit with the number of k-points, etc and the number of CPU. Sometimes having a bit more bands but optimizing the scalability drives you to a much faster calculation.
Did you specify the parameters of paral_kgb yourself (npband, npkpt, npfft, bandpp) or did you use the automatic flag?
If you initialize paral_kgb=1 by hand can you send the values you put?
Best wishes,
Eric

baisxun
Posts: 6
Joined: Fri Mar 30, 2018 5:10 am

Re: mpi issues

Post by baisxun » Mon May 07, 2018 6:01 pm

ebousquet wrote:Dear baisxun,
You can increase nband to >167 that can fit with the number of k-points, etc and the number of CPU. Sometimes having a bit more bands but optimizing the scalability drives you to a much faster calculation.
Did you specify the parameters of paral_kgb yourself (npband, npkpt, npfft, bandpp) or did you use the automatic flag?
If you initialize paral_kgb=1 by hand can you send the values you put?
Best wishes,
Eric


Dear Eric,

Thanks you so much for your reply.

I'm running on 20 processors (just to test mpi with one node on our cluster). The parallel parameters I used are as follows:

paral_kgb 1
npband 4
npfft 2
npkpt 4
bandpp 1

If I understand the parameter settings correctly, it multplies to 32, which is bigger than 20, but it somehow still runs.

So I should assign, for example, 180 processors, which is bigger than 167, and come up with a combination of npband*npfft*npkpt that gives 180, correct?

Best,

Shixun

ebousquet
Posts: 469
Joined: Tue Apr 19, 2011 11:13 am
Location: University of Liege, Belgium

Re: mpi issues

Post by ebousquet » Tue May 08, 2018 5:33 pm

Dear Shixun,
Lets first suppose we parallelize only on bands and kpoints (i.e. npfft=1).
The best is first to have npkpt=nkpt because the parallelism over k-point is the most efficient (it requires low communication between MPI processes).
Then for the bands the deal is to have nband/npband = integer. In your case nband=167 is a really bad number because it cannot be divided by anything but 167... nband=168 is much better because it can be divided by 2, 4, 6, 8 and 21, which can thus be used as values for npband.
Parallelizing beyond npband=16 bands will start to give a less good speedup (this should be tested on your system and machine to see up to which npband value you still have a speedup) and so in your case good values for npband are 2, 4, 6 or 8 if you set nband=168.

The number of CPU of your job should always be equal to npkpt*npband*npfft.

If you have to go further with more CPU numbers, you can start to switch on the parallelism over fft-grid points by having npfft>1.

Another possibility is to use the OpenMP parallelism through bandpp flag (if you have compiled Abinit with OpenMP flag) if you want to use threading, let me know if you want to use it on the top of MPI.

Best wishes,
Eric

baisxun
Posts: 6
Joined: Fri Mar 30, 2018 5:10 am

Re: mpi issues

Post by baisxun » Tue May 08, 2018 7:32 pm

ebousquet wrote:Dear Shixun,
Lets first suppose we parallelize only on bands and kpoints (i.e. npfft=1).
The best is first to have npkpt=nkpt because the parallelism over k-point is the most efficient (it requires low communication between MPI processes).
Then for the bands the deal is to have nband/npband = integer. In your case nband=167 is a really bad number because it cannot be divided by anything but 167... nband=168 is much better because it can be divided by 2, 4, 6, 8 and 21, which can thus be used as values for npband.
Parallelizing beyond npband=16 bands will start to give a less good speedup (this should be tested on your system and machine to see up to which npband value you still have a speedup) and so in your case good values for npband are 2, 4, 6 or 8 if you set nband=168.

The number of CPU of your job should always be equal to npkpt*npband*npfft.

If you have to go further with more CPU numbers, you can start to switch on the parallelism over fft-grid points by having npfft>1.

Another possibility is to use the OpenMP parallelism through bandpp flag (if you have compiled Abinit with OpenMP flag) if you want to use threading, let me know if you want to use it on the top of MPI.

Best wishes,
Eric



Eric,

Thank you! That helped a lot! I am so new at this at I didn't even know nband is something you can manually redefine, how embarrassing...

According to our cluster manager, he did compile ABINIT with OpenMP. So is that why I get the message "you should set the number of threads to something close to 1" ? In the trial runs I either set bandpp = 1 or don't specify anything for bandpp, so intuitively I shouldn't get that message, I suppose?

Best,

Shixun

ebousquet
Posts: 469
Joined: Tue Apr 19, 2011 11:13 am
Location: University of Liege, Belgium

Re: mpi issues  [SOLVED]

Post by ebousquet » Wed May 09, 2018 6:26 pm

Hi Shixun,
Don't be embarrassed, we always have to start from some point ;)
You can thus indeed define nband manually.

OK for the compilation of Abinit with OpenMP. If you don't use the OpenMP parallelization in your job, then don't define bandpp in your input and be sure that the OMP_NUM_THREADS of your system is set to 1 (to enforce that no threading will be used).

If you want to use threading, it is a bit more complicated but the advise is to set bandpp=nband, put npfft=1 (the OpenMP threading is not, yet, compatible with fft parallelization), npband=1 and npkpt as before, i.e. =nkpt. Then you have to set OMP_NUM_THREADS to a value that is compatible with bandpp, i.e. bandpp/OMP_NUM_THREADS is an integer (as before with bandpp and nband in the MPI case).
Then the number of "processors" should be equal to npkpt and the total number of CPU to request will be equal to nkpt*OMP_NUM_THREADS.

Lets suppose we have a system with kpt=10 and nband set to 80 we have the following input flags:
nband=80
npkpt=10
npfft=1
npband=1
bandpp=80

And in a PBS job script you should have something like that (it is an example, but other possibilities are possible that depends on the architecture of the machine you are running on) if we want to run on 8 threads:

...
#PBS -l select=10:ncpus=8:mpiprocs=1:ompthreads=8:vmem=21000mb
...
export MKL_NUM_THREADS=8
export OMP_NUM_THREADS=8

Where here I'm running on 10 nodes for MPI parallelism of k-points (for each k-points that requires low communication) and on each node I request 8 cpu that I all use for threading.

If all of that is quite obscure to you, ask to your machine cluster manager, he should know all about these technicalities.

Best wishes,
Eric

baisxun
Posts: 6
Joined: Fri Mar 30, 2018 5:10 am

Re: mpi issues

Post by baisxun » Wed May 09, 2018 9:47 pm

ebousquet wrote:Hi Shixun,
Don't be embarrassed, we always have to start from some point ;)
You can thus indeed define nband manually.

OK for the compilation of Abinit with OpenMP. If you don't use the OpenMP parallelization in your job, then don't define bandpp in your input and be sure that the OMP_NUM_THREADS of your system is set to 1 (to enforce that no threading will be used).

If you want to use threading, it is a bit more complicated but the advise is to set bandpp=nband, put npfft=1 (the OpenMP threading is not, yet, compatible with fft parallelization), npband=1 and npkpt as before, i.e. =nkpt. Then you have to set OMP_NUM_THREADS to a value that is compatible with bandpp, i.e. bandpp/OMP_NUM_THREADS is an integer (as before with bandpp and nband in the MPI case).
Then the number of "processors" should be equal to npkpt and the total number of CPU to request will be equal to nkpt*OMP_NUM_THREADS.

Lets suppose we have a system with kpt=10 and nband set to 80 we have the following input flags:
nband=80
npkpt=10
npfft=1
npband=1
bandpp=80

And in a PBS job script you should have something like that (it is an example, but other possibilities are possible that depends on the architecture of the machine you are running on) if we want to run on 8 threads:

...
#PBS -l select=10:ncpus=8:mpiprocs=1:ompthreads=8:vmem=21000mb
...
export MKL_NUM_THREADS=8
export OMP_NUM_THREADS=8

Where here I'm running on 10 nodes for MPI parallelism of k-points (for each k-points that requires low communication) and on each node I request 8 cpu that I all use for threading.

If all of that is quite obscure to you, ask to your machine cluster manager, he should know all about these technicalities.

Best wishes,
Eric



Thank you Eric, I'm sure this settles everything.

Best,

Shixun

Locked