Updated stats:

Average Generation: 80.6
Pace (average time to complete a generation): 4.25 days

timestamp_first_indexed generation num_units completed_last_day 
----------------------- ---------- --------- ------------------ 
2019-10-01 22:26:53     000                                     
2019-10-30 18:58:54     001        6                            
2019-12-08 11:56:25     002                                     
2020-01-12 02:02:34     003                                     
2020-02-08 03:43:00     004                                     
2020-02-24 06:27:42     005                                     
2020-03-09 17:38:25     006                                     
2020-03-17 08:44:19     007                                     
2020-03-23 20:52:24     008                                     
2020-04-01 14:39:46     009                                     
2020-04-12 08:29:32     010                                     
2020-04-21 02:41:36     011                                     
2020-05-02 03:16:28     012                                     
2020-05-10 13:29:40     013                                     
2020-05-22 10:46:51     014                                     
2020-06-02 21:07:48     015                                     
2020-06-20 20:53:08     016                                     
2020-07-01 12:31:12     017                                     
2020-07-09 18:39:23     018                                     
2020-07-18 16:08:31     019                                     
2020-07-26 16:32:08     020                                     
2020-08-08 15:15:22     021                                     
2020-08-19 00:49:10     022                                     
2020-08-24 07:02:09     023                                     
2020-08-30 05:56:33     024                                     
2020-09-04 11:35:58     025                                     
2020-09-09 17:27:07     026                                     
2020-09-15 06:25:11     027                                     
2020-09-20 10:01:14     028                                     
2020-09-25 22:07:49     029                                     
2020-10-02 07:08:22     030                                     
2020-10-07 17:55:57     031                                     
2020-10-14 16:25:19     032                                     
2020-10-18 20:05:40     033                                     
2020-10-25 15:34:22     034                                     
2020-10-31 22:55:26     035                                     
2020-11-04 06:29:28     036                                     
2020-11-12 06:33:47     037        1         2                  
2020-11-17 09:21:26     038        2                            
2020-11-24 13:47:28     039        2                            
2020-11-30 07:44:02     040        1                            
2020-12-07 20:20:00     041        2         1                  
2020-12-13 18:26:56     042        2         5                  
2020-12-20 00:33:11     043        7         1                  
2020-12-25 22:27:11     044        3         3                  
2021-01-01 07:57:34     045        6         2                  
2021-01-07 18:08:33     046        7         2                  
2021-01-15 02:41:00     047        5         4                  
2021-01-22 20:25:40     048        7         1                  
2021-01-28 10:53:04     049        3         1                  
2021-02-03 14:32:54     050        6         1                  
2021-02-09 03:20:45     051        4         2                  
2021-02-16 14:14:47     052        6         2                  
2021-02-22 01:22:20     053        9         5                  
2021-02-28 10:29:30     054        7         4                  
2021-03-06 18:23:14     055        7         2                  
2021-03-12 10:16:29     056        8         7                  
2021-03-17 08:30:15     057        12        5                  
2021-03-23 06:08:46     058        8         4                  
2021-03-29 22:39:10     059        7         3                  
2021-04-05 05:01:38     060        8         10                 
2021-04-10 21:09:07     061        19        2                  
2021-04-16 23:20:59     062        12        2                  
2021-04-22 07:50:06     063        9         2                  
2021-04-28 23:02:38     064        7         5                  
2021-05-04 04:45:55     065        11        6                  
2021-05-09 14:11:18     066        10        5                  
2021-05-16 14:55:41     067        16        6                  
2021-05-23 15:02:08     068        15        1                  
2021-05-26 06:43:43     069        19        5                  
2021-05-29 18:38:55     070        16        8                  
2021-06-03 15:46:15     071        25        4                  
2021-06-11 23:13:21     072        15        8                  
2021-06-15 11:54:58     073        22        7                  
2021-06-22 00:30:34     074        25        12                 
2021-06-27 11:56:43     075        43        17                 
2021-07-02 15:06:05     076        105       59                 
2021-07-08 20:49:12     077        447       236                
2021-07-14 07:30:06     078        2000      880                
2021-07-18 14:21:26     079        5171      1667               
2021-07-20 23:37:16     080        7785      1915               
2021-07-23 21:00:51     081        7780      1881               
2021-07-27 02:27:09     082        6350      1250               
2021-07-29 02:04:50     083        3445      584                
2021-07-30 14:32:48     084        1517      263                
2021-08-03 02:15:23     085        538       64                 
2021-08-05 08:06:31     086        71                           

I didn't like the idea of holding 2.5% of workunits before releasing the next leading generation just to keep the spread from getting too wide, so we are only holding 0.5% and will just let the spread be wider. Right now we are designating generations more than 7 behind the lead as stragglers (i.e. at the moment that is generations 78 and earlier). I will leave it here for the next week and see how things go. The next generation or two will arrive quickly but then it is should settle into a pace of about every 4.25 days.

One item of interest - the generations 80-85 (the lead generation) completed 23.0% of the units that were in them as of 24 hours ago while the generations 78 and earlier completed 34.5% of the units that were in them as of 24 hours ago. This is another way of looking how sending the stragglers to reliable hosts is able to complete them faster. Generation 79 completed 28.1% of its results .

They received quite a few additional resources from participants after a recent news posting asking people to increase their commitment in terms of downloaded tasks. They have very little backlog, meaning that very soon after a unit of one generation is validated it goes right out again as a newer generation unit.

It's possible that if anyone has too large of a cache on their computer they could adjust that down to allow some to go to others, but beyond that it looks like most all work units will be in an assigned state from here on out. They could be more aggressive and limit more work units to machines that return quickly to make more available to that particular group but then that runs against their stated goal of not having work units waiting for available resources.

I found the wait to be as long as one hour in some instances. I would run a longer queue so that work will be available to the thread during the "drought". I have removed my machines from ARP1 due to the lack of work. Just like I predicted the lengthening distribution spread, I'm predicting that members will get tired of waiting for work and move to other projects ot sub-projects and then, in about 2 months, the work flow will drop under 15,000 WUs per day. Then they will be asking for more cores again. LIke they say, fool me once......

I'm very confused as to why you think we fooled you.

One of the mistakes that I think volunteer computing has done is created this perception that work will continuously available to run and that there is some form of issue if work is intermittent. Most researchers don't operate this way. It is usually a much iterative process of run a bunch of work, analyze the data, have new ideas and run more work. However, many projects that were first attracted to volunteer computing needed much more computing power that they couldn't get elsewhere so they tended to have enormous needs for computing power and could run continuously for years. This is great and I'm happy that volunteer computing can meet that need. However, there is much larger class of research projects that have a need for a lot of computing - but not at a such a scale that they can provide continuous work. Right now in volunteer computing this tends to be viewed as a "problem" rather than the norm.

WCG with our multiple sub-projects and BOINC as a whole is designed to support this intermittent model. You can set a preferences for multiple projects and if there currently isn't work for one, you will get it from another. That is ok and great. If you think that our African Rainfall Project, ClimatePrediction.net and Einstein@Home are the worthy projects you want to run. Great! You client will rotate through each each BOINC project asking for work and sometimes you will get a job from that project and sometimes not but overall your client will continuously do work for one of them. The same thing could be achieved just within WCG. You could sign up for African Rainfall Project, Help Stop TB and Smash Childhood Cancer (which are all intermittent project) and then check the box for "If there is no work available for the project(s) I have selected above, please send me work from another project.". If there is work for one of your preferred projects, then you will get it. If not, you will get work from Mapping Cancer Markers or OpenPandemics and your client will continuously process work. I do not see why you should unsign up from African Rainfall Project just because work is not continuously available for it.

As far as the pace of the project. The biggest value to the researchers for the African Rainfall Project is in getting the full set of data for the project. The best way to measure how fast the project is running is by how fast the average generation is moving forward. Over the past two months with the work we have been doing on the backend to get the next generation back out as fast as possible as well as recruiting additional volunteers to help accelerate the project we have been able to reduce the time to move the average generation forward from 1 generation every 6 days down to 1 generation every 4.3 days. That is a huge acceleration in the project.

If we had 20,000 jobs sitting around waiting for a client to request them like we had 2 months ago, that would mean that about 1/3rd of the units on the project were just waiting around and not making progress which slows the project. I'm recording the size of jobs available to send every 20 minutes and it generally shows only 100-150 jobs ready to send which is close to as low as we can get. That means that often a request for work for African Rainfall Project will get a job, but not always. This is the way it needs to be to complete the project quickest and get the full data into the hands of the researchers. This has caused the distribution of units between generations to spread out some, but this only matters at the very end where the time between the first unit to reach its final generation and the last unit to reach its final generation might be a little bit longer (maybe 3-4 weeks longer). However, if we have removed many months from the overall length of the project (which we have) then this is a huge gain for the project.

Can you help me understand what it is that I don't understand about why you think we fooled you?

My first unit from generation 086... ARP1_0011268_086_3