Regarding down-time:

Access to the [non-BOINC) database server appeared to be unavailable from about 10:30 (UTC) to about 20:00 (UTC) on 2024-05-02 -- no mid-day stats run, forums inaccessible, other non-BOINC http access getting 500 or 503 errors (so those of us who use the APIs to collect usage stats couldn't do so during that time,)

I've just noticed there was a brief message about this on Facebook (which I can see without an account...) but I've not seen an explanation here. It would be nice to know what broke and (more importantly) why :-)

Forum access is still a bit iffy, but at least it seems to currently be slow rather than broken -- I wonder how long it'll last for this time...

And it took 30+ seconds to let me preview this for the first time, and then it didn't load the style-sheet... The next preview worked but the one I did prior to posting got a 503 error! How can normal usage cause such issues (he asked, plaintively...)?

Cheers - Al.

P.S. I don't do "social media" so I have no idea whether they posted anything on X/Twitter (which I can't see without an account...)

When I looked, right before I went to bed, Facebook said it as 7h (7 hours) ago when they posted the site was down and it was 2h ago when they announced that the system was up again. So there was about 5 hours between the two announcements, was my conclusion then.

Upon checking again, the first message was on "Thursday, May 2, 2024 at 6:21 PM" ("We are aware that the website is currently unavailable. The database server appears to have gone down, and our tech team is working closely with the data centre where our servers reside to resolve this issue. We will continue to provide any updates that we are able to share.") and the second one was on "Thursday, May 2, 2024 at 10:57 PM" ("Hi all, we are happy to share that the website is back up!").
I don't know in what timezone the dates of the posted messages are reported on Facebook, but the first message was 16h ago now and my local time is 10:24 now as I type, so 16 hours ago is 18:24 (my local time). That is 16:24 UTC, when they posted "We are aware …" (and Facebook's time is my local time apparently, but you'll never know ).

According to my own observations, hourly API failures occurred from 11:00 UTC to 20:00 UTC and at 23:00 UTC and 01:00 UTC (this Friday).

I am also glad that the site is back up again.

Adri
PS The site is also very responsive this morning (11:00 local time, 09:00 UTC), as opposed to the past few days(?), when Grumpy Swede wrote "molasses in the system").

I stand corrected.
Thanks

Cheers

I don't think my result list is clearing properly. Is anyone noticing this?? I have some MCMs that had a second valid result come in over 24 hours ago.

I know it isn't about when my WU is validated. It is when the second one is validated. I'm running with no queue at the moment, so I'm usually first.

[Edit: I see Adri has provided the one-line answer whilst I was compiling my post :-) -- ah, well, at least I can confirm what he says and provide some evidence of checking...]

Under normal circumstances I'd expect a validated task to pass through the file deletion process about 24 hours after validation, and the purge to occur a further 24 hours later. Even if assimilation is running as expected, there might still be the odd exception -- with MCM1 look for No Reply tasks that actually return and validate or get flagged as Too Late, as the actual purge seems to take place based on when those atypical results get their final processing, which can be more than a day later :-)

I've had a look at my latest daily stats collection (taken at 02:00 UTC as usual, so around the time you posted), and I've cross-checked the counts of transitions from "merely validated" to "files deleted" and the numbers of purged items, and things seem to tally up in relation to what happened on the preceding day regarding new validations and assimilations.

I've also got a tracker that watches for tasks moving past assimilation, file deletion or purging (run every 15 minutes if the APIs don't fail!) and that seems to show a steady flow of assimilations and purging this morning. If I also use the results pages to count currently validated items then discount items I know have been assimilated I'm not seeing a mass build-up of unassimilated tasks either. I'm therefore tempted to assume there's not a problem at present (I hope I haven't just jinxed it!)

Not only do I collect this sort of data on a daily basis but I produce reports based on what I've got each day and if I see an irregularity I'll flag it up (as I did with the assimilators a while ago!)

Cheers - Al.