Dear volunteers,

We have taken additional measures to increase the quantity of WUs we can send out, and we have been able to increase the quantity of WUs in flight at any given time. Volunteers should see this reflected on their devices now, and perhaps even over this past week.

We are also relieved to share that the hosting data centre has assigned additional personnel on site to resolve our networking issues, meaning a fix is imminent. We will share with you any further updates we receive from the data centre. The network fix will allow us to bring our remaining servers online, stabilizing and further increasing the WU supply.

Thus, until we are able to deploy all dedicated servers, we must continuously adjust and monitor tasks scheduled in Aurora/Mesos to keep the tasks balanced and the workunits flowing, and so far this process is unduly intensive and sporadic. For example, a recurring job may saturate the scheduler by creating a large number of downstream jobs. This flood of new jobs might then throttle the processing rate of other waiting jobs and thereby interrupt the supply of work. To fix the problem, we would need to temporarily deschedule the parent job, decrease its frequency, or decrease the priority of its children in such a way that does not starve other stages of the pipeline.

Last week, we mentioned that we have begun to investigate concerns over statistics, credit, streaks, and database dumps raised by volunteers. We will have an update on some of these issues next week. We also plan to release a more structured breakdown from the tech team similar to a CHANGELOG starting next week or the week after so that we can increase the frequency and clarity of updates.

Future Plans for Aurora/Mesos Replacement by SLURM at the WCG
With the above in mind, although we should be able to immediately deploy additional server resources for Aurora/Mesos job scheduling once networking issues are resolved, our team has greater familiarity and experience with the SLURM scheduler, an alternative to Aurora/Mesos. SLURM is a mature technology currently in use at many of the world’s foremost supercomputing centres, and we intend a full transition to SLURM soon after WCG full restart.

Pending some investigation, we may also look to expand our message-passing layer and implement a publisher/subscriber model and some notion of back-pressure to dictate the chain of downloading data from researchers and creating workunits with which to stock the feeder. From what we have observed, we can expect the move to SLURM will distribute our internal server resources more efficiently than Aurora/Mesos currently does, while losing no functionality. This should be relatively straightforward to port since it overlaps with the existing skill-set of the team.

However, this work is not a higher priority than addressing long-standing concerns of volunteers, which we are finally carving out the bandwidth to address.

Thanks for your patience and have a great weekend!
-WCG Tech Team