help with OMP fortran

General OpenMP discussion

help with OMP fortran

Postby kon » Sun Jun 03, 2012 10:24 pm

Dear Fernando,

Thanks for your reply. I have changed the code as below but I still get a crash. I guess still need your help. Thanks


Kon


The compilation:
/opt/intel/bin/ifort collaeu.f90 -L/opt/intel/composerxe-2011.2.137/mkl/lib/intel64 -I/opt/intel/composerxe-2011.2.137/mkl/include/intel64/ilp64 -lmkl_blas95_ilp64 -lmkl_lapack95_ilp64 -Wl,--start-group /opt/intel/composerxe-2011.2.137/mkl/lib/intel64/libmkl_intel_ilp64.a /opt/intel/composerxe-2011.2.137/mkl/lib/intel64/libmkl_intel_thread.a /opt/intel/composerxe-2011.2.137/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -openmp -lpthread -o collaeu

The code:

program Colleau_algorithm
!$ use OMP_LIB

implicit none
INTEGER :: i, j, k, l, m, n, io, irow, icol, numAnim, ierr, jerr, numOfThreads
REAL(8), DIMENSION(:,:), ALLOCATABLE :: A, Ainv, T, Tinv, Asub
REAL(8), DIMENSION(:), ALLOCATABLE :: D, Dinv, Sol, Rhs, r, v, s, w, y, x
INTEGER, DIMENSION(:,:), ALLOCATABLE :: Ped
INTEGER, DIMENSION(:), ALLOCATABLE :: indx, indx1
real(8) :: a1
real :: start_time, end_time

numOfThreads = 8
!CALL OPM_SET_NUM_THREADS(numOfThreads)
call cpu_time(start_time)

OPEN(1,FILE="/group/adhis/Kon/800Kdata/holstein/old_50K/A.hol")
OPEN(2,FILE="/group/adhis/Kon/800Kdata/holstein/old_50K/Asubset.hol")

n = 0; io = 0
do
read(1,*,iostat=io) i, j, k, a1, l
if(io < 0) exit
if (i == 0.and.j==0.and.k==0) cycle
n = n + 1
enddo
numAnim = n
print*, n, " animals in the pedigree "
ALLOCATE (D(n), Sol(n), Rhs(n), stat=ierr )
if (ierr.ne.0) then
print*," Allocation request for D, Sol, Rhs denied! "
stop
endif
ALLOCATE (Ped(n,3), v(n), indx(n), indx1(n), stat=jerr )
if (jerr.ne.0) then
print*," Allocation request for Ped, v, indx denied! "
stop
endif

rewind 1
n = 0; io = 0
do
read(1,*,iostat=io) i, j, k, a1, l
if(io < 0) exit
if (i == 0.and.j==0.and.k==0) cycle
n = n + 1
Ped(n,1) = i; Ped(n,2) = j; Ped(n,3) = k
D(n) = a1; indx(n) = l
enddo
print*, " second reading : ", n, " animals in the pedigree "
PRINT*, " Commencing Colleau calculations for sub-matrix : "

m = 0
do i = 1, n
if(indx(i) > 0) then
m = m + 1
indx1(m) = i
endif
enddo
print*, " Number of animals in the subset matrix of A is : " , m
ALLOCATE (Asub(m,m), stat=jerr )
if (jerr.ne.0) then
print*," Allocation request for Asub denied! "
stop
endif



!$ call OMP_SET_DYNAMIC(.false.)
!$ if (OMP_GET_DYNAMIC()) print *,'Warning: dynamic adjustment of threads has been set'
!$ call OMP_SET_NUM_THREADS(5)

!$OMP PARALLEL DO &
!$OMP PRIVATE(k,i,j,n,m, Rhs, Sol, v, Ped, D, Asub, indx1)

do k = 1, m
Rhs = 0
Rhs(indx1(k)) = 1.0
Sol = 0
!
do i = n, 1, -1
Sol(i) = Sol(i) + Rhs(i)
if (Ped(i,2) /= 0) then
Sol(Ped(i,2)) = Sol(Ped(i,2)) + Sol(i)*0.5
end if
if (Ped(i,3) /= 0) then
Sol(Ped(i,3)) = Sol(Ped(i,3)) + Sol(i)*0.5
end if
enddo

!
v = 0
do j = 1, n
if (Ped(j,2) == 0 .and. Ped(j,3) == 0) then
v(j) = D(j)*Sol(j)
end if
if (Ped(j,2) /= 0 .and. Ped(j,3) /= 0) then
v(j) = D(j)*Sol(j) + (v(Ped(j,2)) + v(Ped(j,3)))*0.5
end if
if (Ped(j,2) /= 0 .and. Ped(j,3) == 0) then
v(j) = D(j)*Sol(j) + (v(Ped(j,2)))*0.5
end if
if (Ped(j,2) == 0 .and. Ped(j,3) /= 0) then
v(j) = D(j)*Sol(j) + (v(Ped(j,3)))*0.5
end if
end do
do j = 1, m
WRITE(2,FMT='(2i10,f15.5)') k, j, v(indx1(j)
Asub(k, j) = v(indx1(j))
enddo

end do




!$OMP END PARALLEL DO


!WRITE(*,FMT='(" Final subset matrix : ")')

do i = 1, m
do j = 1, m
WRITE(2,FMT='(2i10,f20.10)') i, j, Asub(i,j)
end do
end do

call cpu_time(end_time)
print*, " Elapsed time taken to calculate a subset of A is : ", end_time - start_time


end program
Last edited by kon on Mon Jun 04, 2012 8:33 pm, edited 1 time in total.
kon
 
Posts: 8
Joined: Sun Jun 03, 2012 9:11 pm

Re: help with OMP fortran

Postby ftinetti » Mon Jun 04, 2012 12:13 pm

Hi,

In a quick look I see there are race conditions on assignments to irow and icol. Also, you should be careful in making I/O from threads.

Please include the code between code "tags".

HTH,

Fernando.
ftinetti
 
Posts: 558
Joined: Wed Feb 10, 2010 2:44 pm

Re: help with OMP fortran

Postby kon » Tue Jun 05, 2012 4:27 pm

Dear Fernando,

Thanks for your reply. I have changed the code (see my posting) but still get the crash. Can you help? Thanks.

Kon
kon
 
Posts: 8
Joined: Sun Jun 03, 2012 9:11 pm

Re: help with OMP fortran

Postby ftinetti » Wed Jun 06, 2012 3:42 am

Hi Kon,

Please: 1) do not edit the code so I can see the changes, 2) use code tags, just like:
Code: Select all
  !some code here
  ...


The new code has at least the problem of m as private (i.e. local to each thread) and it is unitialized, so m would not have any useful value (most likely) at
Code: Select all
  !$OMP PARALLEL DO &
  !$OMP PRIVATE(k,i,j,n,m, Rhs, Sol, v, Ped, D, Asub, indx1)
  do k = 1, m             ! a local uninitialized m is created for each thread...
    ...

Best regards,

Fernando.
ftinetti
 
Posts: 558
Joined: Wed Feb 10, 2010 2:44 pm

Re: help with OMP fortran

Postby kon » Wed Jun 06, 2012 5:38 pm

Hi Fernando,

Thanks for your reply. I've used your suggestions(!$OMP PARALLEL DO &
!$OMP PRIVATE(Sol, Rhs, Ped, D, v, Asub, m, n,K,I,J)) but still the program crashes. This is the output:

4483144 animals in the pedigree
second reading : 4483144 animals in the pedigree
Commencing Colleau calculations for sub-matrix :
Number of animals in the subset matrix of A is : 2388
*** glibc detected *** ./collaeuSegmentation fault (core dumped)

I have included l in the privite section but still the result is the same. I really need to make this work because in reality the number of animals in the submatrix is about 30,000.
I will really appreciate your help. Thanks again.

Kon
:cry:
kon
 
Posts: 8
Joined: Sun Jun 03, 2012 9:11 pm

Re: help with OMP fortran

Postby MarkB » Thu Jun 07, 2012 11:42 am

Hi Kon,

You are clearly a bit confused about which variables should be shared and which private!

n,m, Ped, D, and indx1 are all read-only inside the parallel loop: these variables should be shared, since making them private creates uninitialised copies on each thread (as Fernando pointed out).

Asub should also be shared as it is required after the loop: there is no race condition since different threads access different parts of it.

i, j, and k are automatically private since they are loop indices (but declaring them private does no harm).

Rhs, Sol and v should be private: they are temporary arrays which are overwritten on each k iteration.

Try the following:

Code: Select all
     
      !$OMP PARALLEL DO &
      !$OMP PRIVATE(i, j, k, Rhs, Sol, v)
      do k = 1, m           
        ...
MarkB
 
Posts: 408
Joined: Thu Jan 08, 2009 10:12 am

Re: help with OMP fortran

Postby ftinetti » Thu Jun 07, 2012 2:13 pm

Hi Kon,

Maybe this is a minor detail, but I would not use both compiler options

-openmp -lpthread

Why do you use both?

Fernando.
ftinetti
 
Posts: 558
Joined: Wed Feb 10, 2010 2:44 pm

Re: help with OMP fortran

Postby kon » Thu Jun 07, 2012 4:31 pm

Dear all,

Thank you very much for your help.
Let me briefly explain what I am trying to do.
I have a huge matrix of size 4,483,144 x 4,483,144, from which I want to extract a sub-matrix with dimensions 2,388 x 2,388.
Yes, as Mark pointed out, I am a bit confused about which variables should be shared and which private. His notes however made the picture clearer. I’ve made the changes as suggested and the result is the same:

!$OMP PARALLEL DO &
!$OMP PRIVATE(i, j, k, Rhs, Sol, v) SHARED(m,n,Ped,D,indx1)

Output from the program:

4483144 animals in the pedigree
second reading : 4483144 animals in the pedigree
Commencing Colleau calculations for sub-matrix :
Number of animals in the subset matrix of A is : 2388
Segmentation fault (core dumped)

Also I got rid of –lpthread from the compilation.
Note also that in my code I’ve commented out the line
!WRITE(2,FMT='(2i10,f15.5)') k, j, v(indx1(j)
This line was written just to see what is happening and it is not important for the computations. Note also that the program crashes with and without declaration of shared variables.

Kon
kon
 
Posts: 8
Joined: Sun Jun 03, 2012 9:11 pm

Re: help with OMP fortran

Postby MarkB » Fri Jun 08, 2012 2:26 am

Can you test your code for correctness on a (much) smaller dataset first?

You may be running out of stack space, since you are creating some quite large private arrays.
You can set the stack size for the master thread by using ulimit -s (on Unix/Linux), and the stack size for other OpenMP threads by setting
the OMP_STACKSIZE environment variable (you may need to do both: say 200MB should be enough).
MarkB
 
Posts: 408
Joined: Thu Jan 08, 2009 10:12 am

Re: help with OMP fortran

Postby ftinetti » Fri Jun 08, 2012 3:57 am

Hi Kon,

Besides of what MarkB suggests: what is the line in which the program crashes? is it always the same?

Fernando.
ftinetti
 
Posts: 558
Joined: Wed Feb 10, 2010 2:44 pm

Next

Return to Using OpenMP

Who is online

Users browsing this forum: No registered users and 13 guests