What is 'the afternoon'? We are living across the globe, it would help if you specify by UTC, timewise.

Adri

A newsletter just published this evening:

https://www.worldcommunitygrid.org/newsletters

Lots of "Waiting to be sent" tasks now.

One example: https://www.worldcommunitygrid.org/contribution/workunit/641039380

As noted, once again there are lots of "Waiting to be sent" tasks for MCM1; this happens at WCG when a serious backlog of retries for No Reply or Not Started by Deadline tasks gets a chance to build up :-( -- it is more common after outages, but it can happen at other times if there are enough users failing to return work within deadlines.

It seems to be a characteristic of the way work units are being scanned for potential tasks to send out. If precedence is being given to new work (e.g. by scanning work unit numbers in descending order) retries for older work units will be at the wrong end of the queue!

The normal expectation would be that work units are scanned in an order that gives precedence to retries[*1], but I suspect that user complaints about not being able to get work when there was a "retry storm" such as that caused by that rogue cluster (or whatever it was!) a while ago may have influenced the ordering, despite the fact that doing so runs the risk of creating a storm when there doesn't need to be one!

New MCM1 work is being poured into the system for issue so quickly that unless they stop putting new work into the queues for a while this problem will arise sometimes. Over the last year or so, we have then had periods where users are complaining about "Tasks are committed to other platforms" because only retries are available whilst such a backlog is being cleared, and there are so many that the feed buffer is quite likely to only get work for one platform on each load-up.

I wonder if they should simply let retries get out first, or whether a more frequent [but brief] suspension of new work supply should be considered after the next unblock effort? I'm not sure how their work generation tools work, but I suspect something of that ilk could be automated, and if there's no new work for an hour or so every so often it shouldn't concern most users too much given the typical run times for everything except OPNG (which was effectively rationed anyway!)

Cheers - Al.

P.S. If some users could be dissuaded from getting more work than they can possibly run within the deadlines, we'd be less likely to see build-ups of retries at times when there hasn't been an outage, but that's a different matter :-)

*1 -- Ascending order is probably a safe technique under normal circumstances as I believe there is a mechanism for giving urgent work priority despite work unit number order.

I am sure there are a number of ways to address the retry swarms, but not knowing how their feeder system is constructed, I would not even hazard any potential solutions. I would presume the person(s) in charge are probably much better versed than I am in queuing theory and the differences in serial vs, parallel flows, pipeline capacity, differing platforms and other items. Tweaks on these systems without adequate preparation and testing can lead to some unintended consequences. I agree with Al that the caching of too many work units and not being able to complete them in a timely fashion can put added stress on the system. Occasionally it results in some work units being crunched 3 times due to timing issues which is an unnecessary redundancy.
For myself, when the system is operating correctly, I rarely have any problems getting an adequate supply for either Windows or Linux. (No MAC systems here). Whether they are normal units or re-sends is irrelevant to me.

Cheers