4/8/2023 10:56:39 PM | World Community Grid | Temporarily failed upload of OPN1_0129871_01847_0_r1837762973_0: transient HTTP error


Are you kidding me...what the blank now?!??!

My one box (Linux) fails all re-tries, and I'm also unable to get new work units in both Linux and Windows. I'm glad I set a 2 day cache on the Linux box when WCG restarted. I may increase that to 3 days until confidence is restored in WCG availability.

I'm impressed the validator seems to have caught up Friday-Saturday. That's great!

Not impressed at the transient HTTP errors again even on the new flash storage array. Are we out of disk space, or is this just a stuck process?

A self-healing BOINC architecture would be really nice, if such a manual process is simple enough that it can be automated.

I definitely shouldn't have said it !!!!!!!

Almost immediately, I started getting upload errors. Sod's Law in all its glory.

I am pretty sure that WCG is not out of disk space. BOINC upload servers on other projects have reported that they are out of disk space when they are unable to find any free inodes or disk space.

Einstein@home once suffered a situation where it often reported out of disk space when its storage ran out of unused inodes (file system structures used to describe files in Unix-style operating systems), and started taking too long to find used but now free inodes to take uploads with. During that time, it often reported out of disk space errors to BOINC clients. The situation was solved when Einstein@home moved to storage formatted with a later version of its file system that includes a dynamically-built B-tree of used but free inodes so that reuse of inodes that once belonged to deleted files can now be quickly recycled for new files. This is how I know that BOINC upload servers will report out of space errors to its clients if the storage cannot accept an upload.

These errors feel like the earlier WCG download errors where the download server(s) ran out of threads to allow clients to download files, and if you did manage to snag a thread to download a file, the download was super slow. The download side's problems appear to be solved by replacing the HDDs with SSDs. Now the upload side feels like it has the same problems that the download side used to have. When the download side was the bottleneck, there apparently was no way have enough work units in the field to allow result uploads to crush the upload servers. The OpenPandemics and Mapping Cancer Markers projects have very small uploads of several kilobytes that the upload server(s) can easily take. The Africa Rainfall Project has big uploads of around 105 megabytes split into up to 7 files per result by my estimation from what I can see in BOINC's upload page in the advanced view, so these big uploads are crushing the upload server(s).

Well, while the current upload errors might "feel" like the ones we had mainly last year/round the turn of the year, they seem to be distinctively different in what seems to be actually happening.

Back in the past, the system wasn't able to establish a connection and just send a +100 byte header with a respective error message back. No actual data was transferred.

In this weekends sequel however, in my observation over the last few hours (on and off), is that the system actually is transferring data (from host to server(s)), but then either craps out in the middle of the transfer, or right after the transfer has reached 100%, throws an error, which is noticeable on the smaller, just several KB sized files, with those +100 bytes added to the total bytes transferred in the BOINC Manager.

While that was just a waste of available connections in the past, this new behavior is now also significantly wasting bandwidth as there are a lot of multi (2+) MB sized files that get interrupted and retried from scratch each time until successful.

It is really disheartening to see that WCG is caving in to whining of people that they can't crunch this or that (sub) project and release all this just before a long weekend, instead trying this in a more controlled fashion at a time where this can be more closely monitored and in case of issues like currently observed be more timely reacted upon...


Ralf