Thanks for the updates, Mike. The system processed 10 days of data in 1 day. Hopefully this will be done by Monday and we can process some MCM WUs again.

Mike,

Although my earliest MCM WU is from 25 February (by four WU's), I am extremely grateful for you providing these and other metrics.

And to WCG focusing on reducing that backlog to the point where my collection of Valid WU's is finally below 10,000 (from well over 100,000)! That makes the load time MUCH SHORTER when I am reviewing my results.

My opinion of WCG focusing on the backlog (and other aspects) rather than keeping everyone's active queue filled: I wholeheartedly agree with their decision and look forward to the additional project work TigerLily has indicated is planned for release the beginning of this week.

Bruce

My results list is now purged to March 8, and the purging is going faster than ever.

As per bfmorse's post, my "still waiting" dates are usually behind those reported by Mike. However, given when I normally check data, that's not really surprising :-)

I notice that almost all of the results I see that are late compared with Mike's dates tend to be cleared out by the time he posts -- the rest are results where one wingman shows as "Too Late" and the WU wasn't available for assimilation until that flagging happened (even though validated!), or my result validated after the return date and was still waiting for processing by one of the assimilators that seems to be further behind...

Looking at more recent data[*1], purging seems to currently have [significant] precedence over assimilation and so the backlog of items in delete state 2 is being eroded at a high rate; however, at some point the assimilators and file deleters will need to "catch up" once more...

In my case, it has cleaned out over 1700 tasks between 00:00 UTC and 18:00 UTC today, but only 53 Valid items passed through the assimilator in the same time period, with none since 13:00 [still true at 20:00] (akin to a 24+ hour period from early evening on the 20th to early evening on 21st...) I hope that's planned rather than a crash!

At 18:00 I had 943 tasks marked for purge out of about 1350 tasks marked as Valid which (at my normal validation rate) leaves just over two days-worth of results awaiting assimilation. Provided there's another surge in assimilations at some point, we should be back to a natural progression of work from validation to removal by the end of the weekend :-) -- here's hoping that state continues when the challenge starts...

Cheers - Al.

[*1] For the last few days I have been collecting data on results that have passed through the assimilator with a view to spotting when they actually get purged (by watching when they disappear from one of my samples...) -- I didn't start doing this until my backlog got small enough for me to stop feeling guilty about hitting the server, so I can now spot purge times with 15-minute precision (which is probably good enough for this purpose!)

It's exciting to see the quick progress on processing the old WUs. It gives me hope that Monday can be a new start.