I have some that are pending validation although someone else has completed the same WU. In all pending cases with two completions, there is a large disparity in completion times.

A recent addition to my set of BOINC data collection scripts looks for validation issues; a sample item from it is shown below.

ARP1_0014435_129 (WU 155120082) was created 2022-08-25T14:27:14+0000
It has multiple Pending Validation results and none In Progress
First request 2022-08-26T13:24:21+0000;
last return 2022-08-27T20:36:13+0000

The script produces a summary thus (example is for 26th August returns):

Units validated:       6
Units stuck at PVal:   4
Units In Progress:     1

Note that "Units In Progress" refers to items where my result is at Pending Validation but there's at least one other result marked as In Progress...

The total since 26th August (excluding work returned today) is as below (correct as at about 21:30 UTC on 1st September.) :

Units validated:       20
Units stuck at PVal:   23
Units In Progress:     7

It appears that (as at the time of posting) there hasn't been a successful validation for any work unit created after about 13:45 UTC on 25th. All the validated units were older than that (some of them considerably so because of the number of No Reply and Not Started by Deadline tasks ARP1 keeps getting...)

I wonder if there will be a sudden flushing out of one day's worth of these when they've been stalled for 6 days (the same interval as the deadline) -- that is used by the transitioner as an inactivity retry time so they may get a kick then. Whether it'll be sufficient to get them past the blockage is another matter :-(

Cheers - Al.

Mark,

I had the precursor to one of those show up overnight - my success return and one failure return and no retry! And yes, it is related, but tied to the transitioner rather than the validator.

This is not really the place for a [long] "how the transitioner works" piece, so I'll just cite an example of what can happen...

When MilkyWay@home had their disk crash and subsequent excess work unit generation it was common to see work units needing retries that either had "Waiting to be sent" or didn't seem to have a retry readied at all.. What was happening was that there was such a long queue of items waiting for transitioner access that requests to generate a retry and/or submit the retry to the feeder weren't being seen in the usual timely fashion.

The transitioner has a built-in defence mechanism it brings into play at the end of looking at a request; if the new "transitioner time" would be in the past (and yes, if there's a backlog of any type it can happen!) it alters that time to push it into the future; unfortunately, the further in the past the request would have been, the further it shifts it into the future! So if there's a genuine backlog, processed items could be pushed as far as a day into the future (which added to delays when MilkyWay was in difficulties ...)

As far as I know, MilkyWay only run one transitioner, but i suspect WCG run multiple transitioners -- if any of those have crashed and failed to restart, a certain portion of work will not be able to advance, so that could be a reason for problems; otherwise, they may need more transitioners :-)

So it looks as if WCG may need some down time to either clear a transitioner backlog or reconfigure to run more transitioners; turning off various work unit generators instead might help, but it would only be a temporary fix...

And let's just hope that there isn't an over-loaded database and//or file-store access problems at the back of this all - if there is, we may be in for work-unit rationing... :-(

Cheers - Al.

[Edit; to add remark about database/filestore overload...]

I received a few ARP in the last days, together 32. All I sent back between 27. to 29. August are still "pending". Only one is "Valid" (1. Sept, 9:10), two other are pending, wich were sent back on 1. or 2. September.
In the last one or two days i didn't receive any more ARPs.

I don't know if another example helps anything, but I have one here:
ARP1_0021214_128
https://www.worldcommunitygrid.org/contribution/workunit/155155979