Some information which may [or may not] help, regarding assimilation, file deletion and WU purging...

I collect various data from WCG using both the old and new APIs -- whilst the old API only lets me see my results it does tell me the current "file delete state", and I have almost complete records[*1] of "Valid" results for the last several years!

When sifting those records for the last month or so, I note that there was a change of behaviour around the 21st/22nd of October. Prior to that, results that had validated already were being assimilated and purged (and changes in file delete state would often be seen); after that all my results still waiting are stuck at delete state 0, which suggests the MCM1 assimilator is either not running or is having problems!

As quite a lot of my results from as far back as 13th October onwards didn't validate before 21st October I concur with Taurus Oldbull's assessment of how far back the blockage extends. However, the cause is presumably something from 21st/22nd...

Cheers - Al.

P.S. If necessary, I can "show my work" (but there's a lot of it...)

*1 Unfortunately, if there are issues with the API (such as happened between about 13:00 on 2023-10-21 and 15:00 on 2023-10-23 (UTC), there are gaps in that data, so sometimes I may miss something interesting...

[Edited to add comment about backlog date and problem date...]

Purging has started, apparently.
Ten hours ago I asked how many pages of MCM1 with status 'V'alid there were for me:

$ wcgstats -w -sV -aMCM1 -m0 -t3 -rrr
* Let's try to locate the workunit.
Loading results ...
There are 1169 pages of results available.

Ten hours later I did the same:

$ wcgstats -w -sV -aMCM1 -m0 -t3 -rrr
* Let's try to locate the workunit.
Loading results ...
There are 1168 pages of results available.

Since I have a cronjob running hourly to fetch all results, I can compare all my results hour by hour, which ones are new, which ones have disappeared (purged). Since the number of pages shrank during the past ten hours, some results must have disappeared.
So I compared the results and found these ones, (EDIT: in alphabetical order), they have been purged from my results:

< "Name": "MCM1_0206275_0066_1", "ReceivedTime": "2023-10-22T08:51:37",
< "Name": "MCM1_0206275_1688_1", "ReceivedTime": "2023-10-22T09:18:55",
< "Name": "MCM1_0206275_2149_1", "ReceivedTime": "2023-10-18T22:46:36",
< "Name": "MCM1_0206347_2767_1", "ReceivedTime": "2023-10-16T21:18:30",
< "Name": "MCM1_0206347_4208_1", "ReceivedTime": "2023-10-16T21:23:01",
< "Name": "MCM1_0206347_4210_1", "ReceivedTime": "2023-10-16T21:23:01",
< "Name": "MCM1_0206347_6219_1", "ReceivedTime": "2023-10-16T23:25:12",
< "Name": "MCM1_0206358_1343_1", "ReceivedTime": "2023-10-22T13:45:44",
< "Name": "MCM1_0206358_2256_1", "ReceivedTime": "2023-10-17T02:21:02",
< "Name": "MCM1_0206358_3409_1", "ReceivedTime": "2023-10-23T01:17:32",
< "Name": "MCM1_0206358_3430_1", "ReceivedTime": "2023-10-23T01:15:28",
< "Name": "MCM1_0206358_3431_1", "ReceivedTime": "2023-10-23T01:19:41",
< "Name": "MCM1_0206358_3452_1", "ReceivedTime": "2023-10-23T01:25:56",
< "Name": "MCM1_0206358_3456_1", "ReceivedTime": "2023-10-23T01:17:32",
< "Name": "MCM1_0206358_3459_1", "ReceivedTime": "2023-10-23T01:21:44",

Adri

Well, no MCM1 tasks has been purged from my Results list. since October 13.

As Adri noted, there does seem to have been some [very slight] movement on the purge front. However, based on his numbers and what I've seen so far it looks as if it's going to be a long, slow, process :-(

At 02:00 (UTC) today I had 3705 validated tasks waiting, and by 15:50 9 of them had disappeared. However, in the same interval the number of validated tasks waiting went up to 3716 (so 9 out, 20 in!) -- if they continue to validate faster than they get purged we have a real problem...

Cheers - Al.