Unixchick was correct except for 1 minor point.

Unlike many of BOINCs other projects, WCG has never had progress information. Many have asked so we have had to improvise for ourselves. I hope that my daily & weekly reports are of interest.

ARP is very different from any other project in that it computes the weather in 35609 different squares of Sub-Saharan Africa. Each unit computes the weather in that square for 2 days in 24 second steps. Each checkpoint covers 6 hours.

The name of the unit can be broken down into 4 sections. The first is ARP1 for the project. The second is a 7 digit number 0000000 to 0035608 to denote the square we are computing. The third is a 3 digit number from 000 to 183 which refers to the time period (generation) (000 was from 00:00 on 1 July to 23:59 on 2 July). The fourth is a single digit to indicate the copy of the unit that you have.

Each square started in generation 000. As each unit was returned and validated, the result was compared with the weather that actually occurred in those 2 days. If OK, the next 48 hours (generation) would be released.

Because different crunchers took different amounts of time to return their completed copies, the different squares gradually got further and further out of line with each other.

Then we had units for which the computations did not match the actual weather, so they got further out of step. Many of these were cured by reducing their TimeStep from 24 seconds to 18 seconds. However, some have not responded to this adjustment, especally the 3 so-called "ultras" in generations 021 & 022.

This was starting to become a problem not long before IBM handed over to Krembil. I persuaded the IBM tech, to let us have a regular report of the situation. Persuaded was probably the wrong word because he had already got it in mind to do (but might otherwise have run out of time before handover).

This is the main report: https://download.worldcommunitygrid.org/boinc...rp1_stats/generations.txt

After Krembil took over, the situation became worse, but we were able to persuade them to do something about the generation spread. This year, they have been supplying units that were ready to go on a generation by generation basis. This has resulted in workable units closing in on generation 143.

A few days ago I started chasing the 1 remaining moving unit that was still accelerated. I presume that the rest and the extremes are all stuck (759).

The most advanced units reached generation 146 last year.Those within 10 generations had been classified as 'normals', the next 5 as 'accelerated' (132 to 136 and the earlier generations as 'extremes' (104 to 131). The earlier attempts to catch up involved halving the deadlines for accelerated units and also sending 3 copies for extremes. They were already halving the deadlines for resends after the mid-point of the original deadline, so some had 1.5 days and others 3 or 6 days.

Perhaps someone could peer review this and Unixchick put a link on Front Page.

Mike

Mike,

I hadn't answered the original Boca Raton query because I expected a reply from Unixchick or yourself...

In response to your "peer review" comment -- your message seems to be [mostly] factually accurate (and you may be far better informed than I am!)

Details regarding how they decide which generations to process at any given time (and whether it's actually good strategy to bunch everything up) would be opinion-driven![*1]

My questions would relate to what causes particular cells to stall, and what might be done about them. It was my understanding that a lot of the known stalled tasks were ones that had actually failed to produce enough results because the model iteration step was too long to handle the local conditions of the cell -- SIGSEGV or its Windows equivalent was quite a common failure in those cases and they never got as far as validation! Some other tasks undoubtedly failed for other reasons (and a simple re-issue would suffice), and some may have died when validation [or other post-processing?] couldn't succeed.

(You may recall the watch we once had for units that were getting Error returns, with responses from lots of folks, and [eventual] input from Kevin at WCG; that was quite a while ago now!)

As for validation itself -- I had presumed that the first phase of validation was simply the bitwise identity comparison of result files (done, I believe, with an MD5 checksum [or similar]). Whether a check against the expected outcome was then done as a further part of validation and, if so, how close a match was required, is unknown to me (I don't recall seeing it mentioned anywhere, but that doesn't mean it isn't the case!)

I seem to recall reading that Delft used to have to approve cells moving on to the next generation; that would imply it was outside the validation process, but I might be misremembering.

As I said, your original was mostly o.k. as is -- I'm not sure whether any of what I've mentioned might help [without over-complicating things], but it's what I've got :-)

Cheers - Al.

*1 For what it's worth, I think that with the present three groupings of generations it's not good strategy as it will increase the number of units that end up taking 7+ days to resolve because of missed deadlines on "Normal" tasks... I did use the word "opinion" :-)

Mike,

I sometimes wonder whether there's anything in the [ex-IBM] ARP1 toolkit to locate (and explain the reason for) stuck units! Also, it would be interesting to know how trivial (or otherwise) it is to try to re-issue a unit if there's no obvious reason it got stuck in the first place (mis-tagged data, perhaps?)

I think we're all a little frustrated at the lack of that sort of information, but it wasn't exactly forthcoming when WCG had several IBM staff working on it, so it's a bit difficult to expect the current folks to suddenly provide the answers :-)

Ah, well; at least we can process what we get sent... And on that note, it would be nice if more units were released each day when the data centre gets back to full capacity, but we'll just have to wait and see.

Cheers - Al.

Again no movement of extremes. There are 318, all of which appear to be stuck.

No accelerated units validated. 1 is now in generation 136. I think that it is the only accelerated moving out of 442. If anyone has it, please expedite it.

555 normals validated yesterday. There are now 6,419 normals in the generations being released, many of which may be stuck, particularly in generations 137 to 141.

There are now 26,882 held up in generation 143,whjch may start to be released soon as we are now about 95% of the way through generation 142.

There has been an irregular increase in downloads.

Mike

Hi!
1.st one received in 4 days:

ARP1_0034203, so closer to next generation ?

Another 1, late evening:

ARP1_0034472