critical sections embedded within parallel for

General OpenMP discussion

Re: critical sections embedded within parallel for

Postby ftinetti » Tue Apr 15, 2014 4:21 pm

Hi again,

each datafile has data for ~18 variables (position data for ~18 components). And so I am calculating 6 statistics for each of 18 components over all data files. Also, some data files can be small (~kB) and some large (~MBs).


Maybe I'm losing something, but maybe (again...) there is a "little" amount of data involved for O(n) calculations... the I/O would be the bottleneck anyway. If I'm making the math correctly, and depending on the "current" MBs the files have, aprox. 20xMB is still MB or at most a little amount of GB... did you consider to read all the data at once and make calculations (OpenMP threaded, of course) directly on arrays?

HTH,

Fernando.
ftinetti
 
Posts: 582
Joined: Wed Feb 10, 2010 2:44 pm

Re: critical sections embedded within parallel for

Postby armstrhu » Wed Apr 16, 2014 8:47 am

Fernando,
The ~1kB and ~1MB values I just pulled out of the air :P I was just trying to illustrate that the files can differ by orders of magnitude. In general, I cant rely on them being similar in size. My original method did read all of the files into memory then did the analysis on them. However, as I started applying this to larger systems, the memory usage became prohibitively large. So now I have recoded it to read one at a time (in parallel) and add the data to the analysis then release the file from memory.
armstrhu
 
Posts: 9
Joined: Mon Apr 14, 2014 10:56 am

Re: critical sections embedded within parallel for

Postby ftinetti » Wed Apr 16, 2014 8:54 am

Hi

However, as I started applying this to larger systems, the memory usage became prohibitively large. So now I have recoded it to read one at a time (in parallel) and add the data to the analysis then release the file from memory.

then, your current problem is not related to process data in several files but to process the data one file at a time, right? I think that if that's the case, then the load balancing problem should "almost disappear"... but maybe I'm losing something.

HTH,

Fernando.
ftinetti
 
Posts: 582
Joined: Wed Feb 10, 2010 2:44 pm

Re: critical sections embedded within parallel for

Postby armstrhu » Wed Apr 16, 2014 10:34 am

Sorry, I am not wording myself well. My for loop is written to handle one file at a time. But I parallelized the loop, so in reality I am processing num_threads files in parallel.
armstrhu
 
Posts: 9
Joined: Mon Apr 14, 2014 10:56 am

Previous

Return to Using OpenMP

Who is online

Users browsing this forum: No registered users and 9 guests