[TL/DR summary -- assimilations still happening at a good rate, but file deletion and d/b purging falling behind...]

Adri suggests that some attempt at a balance between new work and tidying up after the assimilation problems is being made. It does indeed behave as if the MCM1 WU generator/feeder pipeline is deliberately left unfilled on occasion -- if that is happening, it might explain why I sometimes get sent nothing but retries and at other times retries seem to not get a look in :-)

For what it's worth, I suspect that the only difficulty that sending out new work creates for the post-assimilation clean-up is a slight increase in query times -- the mass successful assimilations will probably be putting an equally heavy load on query times, if not higher, as transitioner queries don't see WUs waiting for assimilation, but as soon as they are assimilated the WUs become visible and may well become part of a transitioner backlog...

By the way, it seems that the file deleter(s) can't quite keep up with the assimilators -- I'm seeing the number of items in delete state 1 (waiting for file deletion) increasing at a fairly steady rate... Also I see a steady increase in items at delete state 2, indicating that the db_purge process seems unable to keep on top of the volume of WUs it needs to process at present, and (unfortunately) there's no real solution to that problem[*1] :-(

(As at 02:30 UTC on 2024-03-06, about 50% of my results assimilated since the 2024-02-29 assimilator(s) restart have been purged; about 40% are waiting to be purged and the other 10% or so are waiting for file deletion.)

As has often been pointed out, a lot of patience is going to be needed over the weeks it is likely to take for the assimilation, deletion, purging backlog to be completely clear. And as has also been said more than once, at least WCG still exists, even if it isn't as resilient as it was when it had access to "free" IBM resources [but in this thread I'm probably preaching to the choir :-)]

Cheers - Al.

*1 I don't know whether db_purge would get through its work faster if there was no other BOINC activity going on! There's a lot of file-writing going on and a lot of database record deletion (and index restructuring) as well... (And no, it isn't possible to run more than one standard db_purge process at once!)

[Edit: added information on %ages purged and waiting.]

Thanks to Adri for spotting the typo :-)

As for not getting work on a particular system... I sometimes find that my Ryzens don't get as much work as my Intel systems if it's mostly retries being dished out. Sometimes the Intels might get a full quota [of retries] whilst the Ryzens get nothing at all :-(

It's probably all coincidental, of course :-)

On a more general issue regarding work issued -- I notice a fair number of new posts in other places bewailing the lack of work, and a check on my systems suggests there might well be more of an issue than previously...

Two of my systems are now out of work and the other two are well below quota. After a pause of several hours, one or two new tasks for WUs created yesterday showed up (well out of sequence!) but there's nowhere near enough to get up to quota...

So it does look as if we're more or less out of work for now (and OPNG supply seems to have dried up too, though that;s not as surprising...) -- reasons as yet unknown :-)

Ah well, with luck it'll help ease the resource pressure on file delete and db_purge.

Cheers - Al.

P.S. there's a wonderful new thread in Website Support about a user not being able to load the results page(s). It turns out the user has nearly a million results; that'll stress the user's browser :-) [and it's an indication of just how much of a total backlog there might be...]

Purging continues. Oldest now 13 November.

Mike

Evidence suggests purging is causing bad communication

Lot's of OPNG tasks coming in here too Adri.