They are far beyond "Have you tried turning it off and then on again".

Uploading, Task Scheduling, and Downloading are unavailable until they resolve the issue.

As one who supports large distributed systems, even the most reliable system with the most redundancy can fail. Almost by definition, it will be something they have never seen before.

While a head-slapper is possible, they are actually rare. It is only the sheer number of systems out there that provide the stories we all read about.

I will thank them for their efforts and for the updates that have provided, and in advance for the updates to come.

[Edit to correct typo.]

I have extended the deadlines for all results on the database that are in progress. Your clients will not see this change, but the backend jobs will read this update when we get those processes back online.

Thanks,
-Uplinger

Thanks for the update, Uplinger. Hope you guys get a handle on things soon. My one and only machine has run itself dry since midnight. That makes me reconsider the size of the cached work I keep from now on.

Cheers!

Edit1: Oh, right, the systems are down so cannot save the changes to my profile. Should've seen that coming.

I'd expect them to use computers big enough that the power cords would use the explosion-proof connectors like those I saw on a computer power cord years ago - those WON'T pull out of the socket unless you first spend a minute or two unscrewing the outer shell.

An idea for how to avoid resends that aren't needed - disable the program that decides what to resend for about 24 hours after uploads are working again.

That's great, thanks!