CPU usage and OpenMP

[pwei]

I have a pretty general question which I didn't find answered in any
forum post yet:

What determines the CPU usage of OpenMP threads?

Background and details:

I have a C application that uses a #pragma parallel for loop to do some
pretty heavy processing that's typically taking several tens of seconds.
I am running it on 64bit Linux (using gcc 4.5.1 or 4.4.6 depending on
the machine) on machines with 8 to 32 cores. While developing it and "in
production" (for the past half year or so) it used to basically saturate
the machine, so that when I checked the CPU usage, I saw as many threads
as there were cores all use 100% CPU, i.e. a total of 1600% CPU usage on
the 16 core machine. I check this using top.

Now, in the last few weeks, I see something different: there are as many
threads as before, but I see a total CPU usage of 100% or 500% or some
n*100% in between, where a few threads are using 100% and the rest of
the threads is getting less CPU. Here is an example top output with 500%
usage (ran on a 16 core machine with OMP_NUM_THREADS=12 set in the
environment):

Code: Select all

   top - 16:00:03 up 384 days, 44 min, 23 users,  load average: 6.69, 3.40, 1.43
   Tasks: 420 total,  13 running, 396 sleeping,   1 stopped,  10 zombie
   Cpu(s): 36.0%us,  0.5%sy,  0.0%ni, 63.4%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
   Mem:     72633M total,    69799M used,     2833M free,      226M buffers
   Swap:     4102M total,      479M used,     3623M free,    64874M cached

     PID USER      PR  NI  VIRT SWAP CODE DATA  RES  SHR S %CPU %MEM    TIME+  COMMAND
    8944 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R  100  3.5   1:18.77 myprogram
    8952 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R  100  3.5   0:29.72 myprogram
    8955 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   99  3.5   0:30.18 myprogram
    8950 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   26  3.5   0:22.84 myprogram
    8954 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   26  3.5   0:23.39 myprogram
    8949 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   25  3.5   0:25.18 myprogram
    8953 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   25  3.5   0:23.33 myprogram
    8947 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   21  3.5   0:22.88 myprogram
    8948 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   21  3.5   0:22.84 myprogram
    8957 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   21  3.5   0:27.32 myprogram
    8956 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   20  3.5   0:22.77 myprogram
    8951 username  20   0 2897m 370m  124 2.8g 2.5g 2868 R   19  3.5   0:22.84 myprogram

My program even prints out run time of the relevant part:

Code: Select all

   51.455s (wall-clock) 131.910s (CPU, 12 CPUs)

showing only a very moderate gain of the parallelization, despite having
12 CPUs available to OpenMP.

The problem is that I don't see anything that has changed, neither in
the code nor in the installation of the machine. The loop in question is
a simple parallel OpenMP for loop with some explicitly shared variables:

Code: Select all

  #pragma omp parallel for default(none) \
          shared(aCube, aPixgrid, cd33, crpix3, crval3, ..., znorm)

I don't restrict the scheduling in any way.

I would be grateful for some hints on what could cause this or how to
debug what's going on.
Peter.

[ftinetti]

Hi Peter,

The problem is that I don't see anything that has changed, neither in
the code nor in the installation of the machine.

Maybe the running environment changed, are there new processes/users there were not before? I see something "suspicious" in the usage of Memory (too close to the limit) and swap (every thread using some swap memory).

HTH,

Fernando.

[MarkB]

Given that it hasn't had one for over a year, a reboot might be worth a try!

[pwei]

Hi Fernando,

Maybe the running environment changed, are there new processes/users there were not before? I see something "suspicious" in the usage of Memory (too close to the limit) and swap (every thread using some swap memory).

The swap is indeed weird, I would have throught that the process is not running long enough to get partly swapped out, especially since there is plenty of RAM just taken up by the cache. But I don't think that swapping happens while the CPU-heavy loop in question is running. I see that the swap increases while I read the data from disk at the beginning of the process, but it doesn't change during the threaded loop. And in my program the bulk of the used RAM is in the shared OpenMP variables. Would that not show up in top for all threads?

Otherwise I don't find memory use special, in fact in production this process takes up to 60 GB of RAM on this machine (even more on another machine that shows the same problem), and still used to run in parallel much more efficiently.

I don't think anything has changed regarding users or special processes. At least I didn't see any other processes that would compete significantly for resources while I run my program.

Given that it hasn't had one for over a year, a reboot might be worth a try!

Good idea, I will try that when I get a chance.

Peter.

[ftinetti]

Hi again,

I don't think anything has changed regarding users or special processes. At least I didn't see any other processes that would compete significantly for resources while I run my program.

Hmmm... then I run out of ideas, at least by now. However, given

MarkB wrote:
Given that it hasn't had one for over a year, a reboot might be worth a try!

Good idea, I will try that when I get a chance.

If you have a chance, try to run your code with no other processes out of those of the OS. BTW, I'm curious about
Processors: type and number
Amount of RAM:
OS:
of your computer.

Fernando.

[pwei]

OK, so rebooting indeed helped! Thanks again for that idea. I now again get

Code: Select all

13.355s (wall-clock) 140.140s (CPU, 12 CPUs)

so a 10.5x speedup.

I run these things on three machines. They all have slightly different hard- and software installed. Curiously, the two machines that show(ed) the problem had zombie processes, the one where I never saw the problem doesn't have any zombies. I wonder if that has anything to do with it...

Fernando, I have forgotten what hardware we ordered exactly, but I can tell you about the properties I see in /proc. One machine has Xeon E5520, 8 cores total (16 if you count hyperthreading) and 72 GB RAM; another has Xeon E5645, 24 cores including hyperthreading and 24 GB RAM; the last one has Opteron 6134, 32 cores including hyperthreading, and 256 GB RAM.

Cheers,
Peter.

[MarkB]

OK, so rebooting indeed helped! Thanks again for that idea.

You're welcome!

I run these things on three machines. They all have slightly different hard- and software installed. Curiously, the two machines that show(ed) the problem had zombie processes, the one where I never saw the problem doesn't have any zombies. I wonder if that has anything to do with it...

As I understand it, zombie processes are supposed to be harmless and not consume and system resources (e.g. CPU, memory). You might have had orphaned processes which did not release their memory, causing your running threads to swap???

[ftinetti]

Fernando, I have forgotten what hardware we ordered exactly, but I can tell you about the properties I see in /proc. One machine has Xeon E5520, 8 cores total (16 if you count hyperthreading) and 72 GB RAM; another has Xeon E5645, 24 cores including hyperthreading and 24 GB RAM; the last one has Opteron 6134, 32 cores including hyperthreading, and 256 GB RAM.

Thanks for the details of the computers, which one is used for

13.355s (wall-clock) 140.140s (CPU, 12 CPUs)

?
(I guess it is on the two Xeon E5645s, though)

Thanks again,

Fernando.

[pwei]

As I understand it, zombie processes are supposed to be harmless and not consume and system resources (e.g. CPU, memory). You might have had orphaned processes which did not release their memory, causing your running threads to swap???

The swapping is apparently normal, even now when I get full speed again, I see this. And most of the RAM is still unused...

Thanks for the details of the computers, which one is used for

13.355s (wall-clock) 140.140s (CPU, 12 CPUs)

?
(I guess it is on the two Xeon E5645s, though)

Sorry for the confusion. That was the first one with E5520s, but run with OMP_NUM_THREADS=12. I found that for this machine and what I'm doing this 12 threads give the optimum runtime.

Peter.