Has any thought been given by Krembil to improving the reliability, stability, and overall production-readiness of the various components of the application and its associated hardware and network components? It seems that there is little redundancy overall (although I was pleased to hear that drive(s) were configured via RAID (hopefully at least RAID level 1 (mirroring)). Also, when an issue occurs late in the week, nobody is available apparently to investigate/resolve the issue over the weekend. It feels like a configuration overall where errors and downtime are resolved without necessarily a sense of urgency. The instability and downtime leads to frustration and a lack of faith amongst at least some of us who donate our computer resources to supporting World Community Grid.

I am glad that there are alternative medical/biology-related distributed computing options available (e.g., Folding@Home and various BOINC-related projects such as Rosetta, TN-Grid, Denis@Home, and SiDock@Home).

Go back 4 weeks and plant the idea to do a test of backing up the project to a spare unit. So 2 weeks to work out the bugs and them BAM, the main server goes down...well, that's mightily convenient we were doing a rest run of backup system...

Then we'll just grouch instead of bringing pitchforks and torches.

Because it's not a RAID problem, but a problem with accessing the controller in the first place. And one problem with RAID5 is, if you have a degraded array, by today's drive sizes, it takes forever to get in sync again.

Under IBM still, there was at some point some change to ZFS as the underlying file system. And that didn't go quite smoothly (though only a couple of days delay IIRC) but it seems that the system currently deployed at WCG is running on a more "traditional" file system.

Ralf

Dear all - I really do appreciate your patience and your thoughts and suggestions. Indeed, many of them we have thought about and some of them we were able to implement. However, as posted before - we do not have the budget IBM was able to put into WCG - we are working on improving funding as we go.

Unfortunately, it was not "just" a disk failure - that would be fine - and it was not "only" the RAID controller failure - the bus failed -- so the whole storage unit failed.


We are grateful for your contribution of computing resources. Some of you may also be able to help WCG by introducing us to foundations or companies that may be interested to become a funding or contributing partner. We have tried and continue trying to form some form of partnerships with Dell, Lenovo, HPE, ... nVidia, AMD, Intel ... AWS, Google, -- that may help us (backend of WCG) but also projects (e.g., ARP) solving equipment challenges (and insufficient government funding for science). Cold calls do not get us far -- but opening the door may help tremendously.


Redundancy for safety and performance would be marvelous. When we were running the HCC project - funded by NIH - we had 4 backups at 3 geographically different locations on three different media. Call us paranoid - but many stories about how individual media looked great at the beginning, only to be very, very obsolete quickly.


Brief update from today:
Database filesystems are up and the data are backed up (tape). Databases are up on the temp-replacement DSS7000 storage system. Science remains down. We are going to create a copy of the science file system while we try to repair it. The RAID controllers seemed to not know about failed disks after restart or swap, and would rebuild RAIDs back to their original configuration.



Thank you
Igor

Ps. And as a reminder - please note, as we described multiple times, WCG owner is UHN, as a legal entity - they do not support WCG in any form. My group is a research lab within Krembil Research Institute, UHN, and we run WCG, as any other research project. We have to find funds for its operation. Note that WCG servers are not at UHN either, as most of my lab servers are also at Sharcnet and Scinet. Thus, it is not appropriate to blame Krembil or UHN - as it would be as accurate as blaming Toronto or Canada.

Our mission remains: Accelerating science by creating a supercomputer empowered by a global community of volunteers. THANK YOU.

well - there is optics and politics and there is reality.

I try to minimize it - but I get "reminded" that UHN needs to be mentioned.

Foremost - WCG is a volunteer platform, transforming open science through global collaboration, citizen engagement and youth outreach. My goal is to push that more (even if we cannot remove the other; at least not now).