We generally get the work from the researchers in "Batches". Our estimate of how much longer a project has is based on the number batches remaining divided by the recent average of batches completed.

When the batches process on the grid a small number of tasks within the batch (0.1% or so) will have "bad luck" and will wind up error outi not because the task is flawed but because it happens to get sent to a combination of computers that either don't return it or error out on it. We automatically re-run these.

We re-worked the process of loading work into BOINC a number of months ago in order to run it within our Apache Mesos/Apache Aurora infrastructure. However, the logic to schedule the re-run of those errored tasks had a flaw and these were not getting picked up and loaded on a timely basis. As a result, the number of batches "in progress" had jumped rather significantly. We fixed that error on Tuesday morning (sept 11). The fix worked really well. Here is the result of that:

batches	date_completed
20	2018-09-13
208	2018-09-12
30	2018-09-11
3	2018-09-10
1	2018-09-09
2	2018-09-08
7	2018-09-07
5	2018-09-06
3	2018-09-05
4	2018-09-04
6	2018-09-03
3	2018-09-02
5	2018-09-01
4	2018-08-31

As you can see we had a huge flurry of batches completing (260 in the last 48 hours vs a normal pack of about 10). As a result, this is has temporarily thrown off our "end of project" completion estimates since those are based on the number of batches completed.

The estimates will return to normal once this surge is over.

+1

Thanks for the quick and detailed reply. Great to get a peek into the back end to understand the jump in completion estimate.

Thanks for the update. I figure I will hit the 2Y Sapphire mark around the end of November 2018 and was sweating it before reaching this forum if I was all of the sudden close to running out of work just before getting to the finish line.