Inconsistency of number of processors in openmp

option, parallelism,...

Moderators: fgoudreault, mcote

Forum rules
Please have a look at ~abinit/doc/config/build-config.ac in the source package for detailed and up-to-date information about the configuration of Abinit 8 builds.
For a video explanation on how to build Abinit 7.x for Linux, please go to: http://www.youtube.com/watch?v=DppLQ-KQA68.
IMPORTANT: when an answer solves your problem, please check the little green V-like button on its upper-right corner to accept it.
Locked
sheng
Posts: 64
Joined: Fri Apr 11, 2014 3:44 pm

Inconsistency of number of processors in openmp

Post by sheng » Thu Sep 04, 2014 5:15 pm

Lately when I scan through the log file I found that peculiarity of Num_procs in OpenMP parallelism.
For example, the command I enter is mpirun -np 4 which suppose to use 4 processors. But I discover the figure in openmp is different:

==== OpenMP parallelism is ON ====
- Max_threads: 2
- Num_threads: 2
- Num_procs: 2
- Dynamic: F
- Nested: F

However the indication stated at the start of dataset is correct
== DATASET 1 ==================================================================
- nproc = 4

I tried with other values but still with same discrepancy:
mpi 2, openmp 1
mpi 5, openmp 8

This has not happened before when I did the tutorial (if I am not mistaken).
Any idea what could possibly goes wrong?

Jordan
Posts: 282
Joined: Tue May 07, 2013 9:47 am

Re: Inconsistency of number of processors in openmp

Post by Jordan » Thu Sep 04, 2014 9:06 pm

The command mpirun only tells you how many MPI processes you want.
Each MPI process can use a given number of OpenMP threads so the total number of threads you actually use is nprocs*Num_threads
For example, on a computer with 8 cores, you can either launch 2 MPI processes with 4 OpenMp threads each, or 4 MPI processes with 2 OpenMp threads. That would be the recommanded use.

The number of MPI processes (nproc) is given to mpirun with -n or -np, both are equivalent.
The number of OpenMP threads (Num_threads) for each MPI process is given by the environment variable OMP_NUM_THREADS. By default the value is usually the number of cores of your cpu (Num_procs).

If you don't configure correctly MPI and OpenMP, you can overload your cpu. For example, if your cpu has 4 cores and you launch a mpi job with -np 4 and auto OpenMP detection, you might have a moment during the execution when you have 16 threads at the same time. It can sometimes be what you want to do if threads have limitated load due to other operations (I/O, locks,...) but for intensive calculation you just want one thread per core, ie nproc*Num_threads=number of cores available.

On supercomputers, you usually need to request either the number of mpi and the number of threads per mpi or the total number of cores you want and use the mpirun to set the mpi processes on all nodes (on recent OpenMPI you have --npersocket --npernode and so on to manage the repartition of the processes) and set the OMP_NUM_THREADS variable.

Cheers

Jordan

sheng
Posts: 64
Joined: Fri Apr 11, 2014 3:44 pm

Re: Inconsistency of number of processors in openmp

Post by sheng » Fri Sep 05, 2014 4:38 am

Thanks Jordan for your quick and extensive explanation.

By default the value is usually the number of cores of your cpu (Num_procs).

The number of cores of my cpu is 8, then I suppose Num_procs should 8 be default, but then this is not the case stated by abinit by auto detection:
mpi=4, OpenMP Num_procs=2
mpi=2, OpenMP Num_procs=1
mpi=5, OpenMP Num_procs=8

Is the OpenMP Num_procs important or I just need to take care number of OpenMP threads (Num_threads) and ignoring Num_procs?

Thank you.

Jordan
Posts: 282
Joined: Tue May 07, 2013 9:47 am

Re: Inconsistency of number of processors in openmp

Post by Jordan » Mon Sep 08, 2014 4:23 pm

What's your cpu model/reference ? Do you have HyperThreading ?

The function omp_get_num_procs (used to print Num_procs) returns the number of online cpus.
My hypothesis is that some of you cores are asleep when you just submit few MPI processes to save energy.

In any case, if you impose OMP_NUM_THREADS to the value you want, you should see the correct value of Num_threads and the code should run smoothly. As I mentioned, if you have Num_threads > Num_procs that would mean that you have an overload. But if you know exactly what you do (meaning you know your computer), you're fine. When the OpenMP starts, the asleep cores will wake up to work and I guess that a new call to the omp_get_num_procs should be 8 at that time.

You can check with the program htop or any other utility (gnome-system-monitor) that all of your cores work.

If you wanna dig deeper, you can write a small code with only one line : write(*,*) omp_get_num_procs() and compile it with your favourite mpi compiler.
Then run the code with 2,4,8 MPI processes and see if you got something like abinit or not.

Code: Select all

PROGRAM testOMP

  USE OMP_LIB

  IMPLICIT NONE

  WRITE(*,*) "Num procs", omp_get_num_procs()

END PROGRAM testOMP


mpif90 -fopenmp for gfortran or mpif90 -openmp for intel

Cheers

Jordan

Locked