Abinit +intel compilators wierd issue while opt geom [SOLVED]

[roginovicci]

I have a vary strange situation. On the same machine (2 xenon e5-2340 processor based) I've compiled abinit 7.8.1 with both intel and gcc compilators. The configuration is as follows:
For intel (intel mpi used):

Code: Select all

export FC=$MPICH/bin/mpiifort                                                                                                                                            
export CC=$MPICH/bin/mpiicc                                                                                                                                              
export CXX=$MPICH/bin/mpiicpc                                                                                                                                            
./configure --prefix=/opt/abinit-$VER --enable-mpi --enable-mpi-io \                                                                                                      
 --with-linalg-incs="-I$INTEL_COMP/mkl/include" \                                                                                                                         
 --with-fft-flavor="fftw3-mkl" --with-fft-incs="-I$INTEL_COMP/mkl/include/fftw" \                                                                                         
 --with-fft-libs="-L$INTEL_COMP/mkl/lib/intel64 -lfftw3xf_intel" \                                                                                                        
 --with-linalg-flavor="mkl" \                                                                                                                                             
 --with-linalg-libs="-L$INTEL_COMP/mkl/lib/intel64  -Wl,--start-group  -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -liomp5 -lpthread -limf -lsvml -lirc" \                                                                                                                                                                       
 --with-linalg-incs="-I$INTEL_COMP/mkl/include" \                                                                                                                         
 --with-dft-flavor="atompaw+bigdft+libxc+wannier90" 

For gcc (mpich used):

Code: Select all

./configure --prefix=/opt/abinit-$VER --with-mpich=$MPICH2 \                                                                                                      
 --with-dft-flavor="atompaw+bigdft+libxc+wannier90" 

And while optimizing the structure with abinit compiled by intel compilators it fails at some step, while optimizing with gcc abinit it finishes smooth. Here is the input I have:

Code: Select all

 acell 5.88 5.88 57.1
   angdeg 90.000  90.000 120.000

rprim sqrt(0.75)  0.5  0.0
     -sqrt(0.75)  0.5  0.0   
      0.0         0.0  1.0   
   ntypat 3                 
   znucl 31 7 13             
   natom 24                
   typat 1 2 1 2 1 2 1 2 1 2 3 2 3 2 3 2 3 2 3 2 3 2 3 2      
xred    
1/3   2/3     0.0000000000
1/3   2/3     0.0637500000
2/3   1/3     0.0841584158
2/3   1/3     0.1479084158
1/3   2/3     0.1683168317
1/3   2/3     0.2320668317
2/3   1/3     0.2524752475
2/3   1/3     0.3162252475
1/3   2/3     0.3366336634
1/3   2/3     0.4003836634
2/3   1/3     0.4207920792
2/3   1/3     0.4845420792
1/3   2/3     0.5049504950
1/3   2/3     0.5687004950
2/3   1/3     0.5923283505
2/3   1/3     0.6536028603
1/3   2/3     0.6732187925
1/3   2/3     0.7344933023
2/3   1/3     0.7541092345
2/3   1/3     0.8153837443
1/3   2/3     0.8349996764
1/3   2/3     0.8962741862
2/3   1/3     0.9158901184
2/3   1/3     0.9771646282

 ixc       7              # Teter XC functional
   ecut      25              # Kinetic energy cut-off (in Ha)
   ecutsm    0.5
   dilatmx   1.25


   kptopt    1

   ngkpt     6 6 4
   nshiftk   1
   shiftk    0.0 0.0 0.5

   iscf      5      # Pulay mixing (default for norm-conserving PSP)
   nstep     150    # Modify default

   ionmov 3
   ntime 150

   toldff    1.0d-4
   tolmxf    1.0d-3
optforces 1
  optcell 0
use_gpu_cuda 0
AUTOPARAL=0

ABINIT fails with warning

Potential-based CG line minimization not converged after13 restarts.
Action : read the eventual warnings about lack of convergence.
Some might be relevant. Otherwise, raise nband. Returning
src_file: scfcge.F90
src_line: 375

Moreover if I use abinit compiled with intel compilator on another machine (cluster) it whent smooth also. The optimization process goes absolutle the same way up to 4th step 5th step differs and the 6th fails.

There is somthing different with reduced forces they all goes to zero in basis at the fifth (pre fails) step of optimisation.

Code: Select all

Cartesian forces (fcart) [Ha/bohr]; max,rms= 9.03107E-03 2.51165E-03 (free atoms)
  0.00000000000000E+00  0.00000000000000E+00  1.54667733343065E-03
  0.00000000000000E+00  0.00000000000000E+00 -2.06600109156256E-03
  0.00000000000000E+00  0.00000000000000E+00 -9.83194404385286E-04
  0.00000000000000E+00  0.00000000000000E+00 -4.16727320416967E-03
  0.00000000000000E+00  0.00000000000000E+00  1.68339746419727E-03
  0.00000000000000E+00  0.00000000000000E+00 -1.72370601891689E-03
  0.00000000000000E+00  0.00000000000000E+00  5.67865482140431E-03
  0.00000000000000E+00  0.00000000000000E+00  4.59214632078102E-03
  0.00000000000000E+00  0.00000000000000E+00  4.48469221935451E-03
  0.00000000000000E+00  0.00000000000000E+00 -1.97601745013876E-03
  0.00000000000000E+00  0.00000000000000E+00  8.04770880797129E-03
  0.00000000000000E+00  0.00000000000000E+00  6.86837849770853E-03
  0.00000000000000E+00  0.00000000000000E+00 -1.29830590065030E-03
  0.00000000000000E+00  0.00000000000000E+00 -9.03107255582250E-03
  0.00000000000000E+00  0.00000000000000E+00  8.13504889824362E-03
  0.00000000000000E+00  0.00000000000000E+00 -2.12193114733188E-03
  0.00000000000000E+00  0.00000000000000E+00 -5.56983517030434E-03
  0.00000000000000E+00  0.00000000000000E+00 -4.53171312473877E-03
  0.00000000000000E+00  0.00000000000000E+00  5.62000940042989E-04
  0.00000000000000E+00  0.00000000000000E+00 -1.60093617041179E-03
  0.00000000000000E+00  0.00000000000000E+00  9.81585503490430E-04
  0.00000000000000E+00  0.00000000000000E+00 -2.38942398773797E-03
  0.00000000000000E+00  0.00000000000000E+00 -8.33702492019031E-04
  0.00000000000000E+00  0.00000000000000E+00 -4.28717808843490E-03

While in previous step its of order -4.30318458033374E-33

Maybe there is somekind of precision issure? Is there paramiter I should pass to intel compilator to rise precision of calculations.

In cluster (Xeon Processor X5550) where abinit compiled by intel the compilator set is 2013 upd 1. I was trying the set in my machine (Xenon E5-2340) to compile good abinit binary with no succes, but I can use the binary compiled in cluster on my E5-2340 based computer with full success. I've tried to use mpich and openmpi with no success also. Maybe I should add some additional flags to intel compilator?

[jbeuken]

Hi,

which version of ifort ?

here , a part of my configure working with ifort 13.1.3 ( without parallel MKL ) :

Code: Select all

--with-fft-flavor="fftw3"  \
--with-fft-libs="-L/opt/intel/mkl/lib/intel64 -Wl,--start-group  -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group"  \
--with-linalg-flavor="mkl"  \
--with-linalg-libs="-L/opt/intel/mkl/lib/intel64 -Wl,--start-group  -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group"  \

can you try this ?

regards

jmb

[roginovicci]

I've tried 13.0.1 (2013 update 1) and 14.1.3 (sp1 upd3) and 14.0.1 I suppose. It all does not seems to be a compilator version issue. But the issue of platform I am using to compile the code. Yesterday I have compiled the code on the same distro (centos 6.5) with the same intel compilator on my home slow machine based on pentium G2120 processor. Compilation was smooth and the binary abinit file works just fine on my Xenon E5-2340 based machine. So I don't think my configure script is bad. It produces good binary compiled in other machine than Xenon E5-2340. It is very very strange. The distro is the same the compilators is the same also but compilation process makes different results for different CPU. Maybe I should reinstall OS to make sure it is not software problem, but I am not ready for that. There is a lot of setups I've already made. Thx in advance, anyway!

p.s. While searching arround I have found my swap partition was not attached to system, reattach and start compilation procedure again. Not sure this is a problem (I have 16Gb of memory installed), but will try.
p.p.s. Forgot to say that I was trying to compile without mkl at all, using internal abinit math library. And tha same result here, it fails at fifth or sixth step.

[Jordan]

The intel compiler for intel cpu can cause some issues with advance intel cpu features (sse/avx)
I have recently experienced this kind of problem with the latest abinit with intel14 on avx and avx2 cpus.

Can you try to compile abinit on your faulty system with absolutly no optimization at all and no "cpu dependant" features (check there is no -march or -mtune option when compiling)?

In addition to that, if you have exactly the same OS and libraries (and same directories tree) you can try to run abinit compiled for your PC on the E5 (assuming there no special instruction used by intel).


Jordan

[roginovicci]

The intel compiler for intel cpu can cause some issues with advance intel cpu features (sse/avx)
I have recently experienced this kind of problem with the latest abinit with intel14 on avx and avx2 cpus.

Can you try to compile abinit on your faulty system with absolutly no optimization at all and no "cpu dependant" features (check there is no -march or -mtune option when compiling)?

Sure I can, but how? I do not add -march -mtune by myself. There is some script required (not strongly indeed) to run before compilation compilervars.sh, iccvars.sh and so on. Should I check them to figure out if the one adds -mtune?

In addition to that, if you have exactly the same OS and libraries (and same directories tree) you can try to run abinit compiled for your PC on the E5 (assuming there no special instruction used by intel).
Jordan

Well, I'am telling you, compiled binaries on pentium G2120 processor platforms works like a charm on E5. Actually there is no need in the same directry structure. The only dependence is glibc version. The other dependencies is intel one. And binaries from the other cluster with xenon X5670 work on E5-2630 just fine too. And that is really wierd.

[Jordan]

Sure I can, but how? I do not add -march -mtune by myself. There is some script required (not strongly indeed) to run before compilation compilervars.sh, iccvars.sh and so on. Should I check them to figure out if the one adds -mtune?

Juste be sure that in your configure file *.ac you have enable_optim=no or --enable-optim="no" in your command line to disable any kind of optimization (testing purpose). At the end of the configuration you should read "optimizations : no" in the summary of important options.
During compilation you should see something that looks like

Code: Select all

mpif90 -DHAVE_CONFIG -I. -I../-O0 -g  file.F90 -c -o file.o

If you do not see -O0 it means you use -O2 (default value) and thus ask for optimizations)

For speed up compilation purpose, also remove the dft_flavors. Can you try to compile with both gcc and intel compilers using the mkl for linalg and fftw (I use fftw3 instead of fftw3-mkl even if I use the mkl implementation) ? (You might need to use --enable-zdot-bugfix for gcc+mkl if a segfault occures when running the code).
You will end with two identical abinits and NO optimizations. Compare both. If they behave the same, compile again removing the --enable-optim or set it to "yes" for both compilers. Compare the new version generated by gcc with the previous one and make sure they give the same result. Now compare with the new intel generated abinit.

This is just to figure out where the problem is.

If I'm not clear, let me not.

Jordan

[roginovicci]

It seems I should make my apologies to you. For some odd reason the one single symbol was lost in my script. And the one is dot. Thus my environment was not changed while compilation process. What I am saying is:

Code: Select all

. ${INTEL_COMP}/bin/compilervars.sh intel64

Has transformed to

Code: Select all

${INTEL_COMP}/bin/compilervars.sh intel64

That is critical error for some xenon processors in fact. Now I'm going to recompile abinit and recheck the functionality, I am pretty sure the problem would be set as solved.


But talking about optimization... Hmmm I really not sure lowering optimization is a good idea. It is probably better to increase optimization level to -O3 to produce more optimized code for the processor and increase precision or whatever. I'm not sure this is the case, just my thoughts. But here is some small question. Compiling abinit with higher optimization level -O3 fails at fallback step. Is it possible to configure abinit to use -O2 for fallbacks and -O3 for main abinit code?
And Thank to anyone in the thread for help!

[Jordan]

Hi

Hope that the missing "." solve your problem.
But if you were not sourcing the intel compiler, what compiler did you use ??? So I guess there was a conflict between compilation time and runtime dynamic libraries...

Anyways.
Of course turning on optimizations is better for production, but it can be convenient for debugging purpose to turn them off.
So yes you can use -O3.
what fallback fails to compile with -O3 ?
You can use any of
fcflags_opt_algo=''
fcflags_opt_atompaw=''
fcflags_opt_bigdft=''
fcflags_opt_etsf_io=''
fcflags_opt_fox=''
fcflags_opt_libpspio=''
fcflags_opt_libxc=''
fcflags_opt_linalg=''
fcflags_opt_netcdf=''
fcflags_opt_wannier90=''
fcflags_opt_yaml=''
to change the optimizations used for each fallback.

Hope that helps.

Jordan

[roginovicci]

But if you were not sourcing the intel compiler, what compiler did you use ??? So I guess there was a conflict between compilation time and runtime dynamic libraries...

Using ldd command I have found the binary was linked to intel libraries only. no libgfortran in the list.

what fallback fails to compile with -O3 ?

atompow I suppose. Don't remember for sure. I've compiled abinit without fallbacks and with O3 optimization level. And started geomopt procedure to if there is a difference to O2 optimized binary.

You can use any of
fcflags_opt_algo=''
fcflags_opt_atompaw=''

This is all lower case variables I should set with export? Smthg like:

Code: Select all

export fcflags_opt_algo=''-fp-model precise -fp-model source -O2"

./configure --prefix=/opt/abintit ....

UPD. I have compiled source code without fallbacks with -O3 optimization and found no difference with the code compiled with -O2 thus I dont see the reason to use -O3. I've posted my configuration script in neighborhood topic, just in a case.