Just got ARP1_0001465_80_1

Thanks to everyone who has upped their contribution to the project. Following the article last week you have reduced the backlog of results ready to send from over 15,000 down to about 2,500. This has been a very nice improvement and at the moment we are running at a pace to complete a generation every 5.1 days. Thanks to everyone running the project!

Due to the decrease in backlog, we will see a bit of spread in the distribution of jobs across the leading generations. This means that over the next week or two we will see the next generation emerge faster. For example, the first generation 79 workunit was created at 07/18/2021 14:21:26 UTC and the first generation 80 workunit was created at 07/20/2021 23:37:16 (about 2.5 days later). It looks like we will see the first generation 81 workunit get created in in the next 24 hours which will only be 3 to 3.5 days since gen 80 (note that it will take a little bit for the first copies of it to be sent out since the front generation always has the lowest priority). This should stabilize at some point (where it stabilizes will depend on the size the backlog).

As noted, there are a total 35,609 units being analysed (which yield about 2.2 results per day for users to process) and we are currently completing about 7,000 units/day (15,000 results/day) on average. 35,609/7,000 = 5.087 days per generation which is what my estimate is based on.

Mike,

Your estimates and predictions have been very accurate - I've been enjoying watching you track the progress and pace.

Here is the latest distribution.

count   generation
6	001
...
2	032
2	033
1	034
0       035
1	036
1	037
3	038
6	039
2	040
7	041
2	042
11	043
8	044
3	045
10	046
5	047
5	048
6	049
10	050
7	051
5	052
10	053
15	054
7	055
10	056
8	057
14	058
7	059
14	060
6	061
13	062
9	063
17	064
17	065
17	066
19	067
21	068
15	069
28	070
21	071
39	072
61	073
196	074
1185	075
3450	076
9398	077
10443	078
6956	079
3334	080
176	081

The stragglers were due to an error that we had to do some work to resolve and get moving again. This is resolved now and the stragglers are catching up. 6 of them unfortunately have to be re-run from the very start due to a combination of failures (we have resolved those failures and we are making sure that this does not happen again).

Outside of the 6, the lagging generation has moved from generation 29 on July 17 to generation 32 so the laggards are catching up (i.e. the tail moved forward 3 generations while the front moved forward 2 generations). We expect this to continue and over time we hope to see only a spread of 6-8 generations between the front and the tail (although the 6 will take longer to catch up)

The backlog had been steadily decreasing from the 15k down to the 2.5k I mentioned above. It dropped down to as low as 2k but it has started to creep back up over the last 18 hours and we are back up to 2.5k. Over the past 48 hours workunits have been completing at a pace that would let us get down to completing a generation every 4.5 days. Given that there is still a backlog we should be able accelerate this even faster. If we continue to see the backlog grow we will attempt to nudge a few more people to increase their allowed jobs on their devices up from 1. We had originally planned to post the news article and then send an email to volunteers already participating in ARP1 but the response to the article itself was bigger than we expected so we have held off on sending the email until we see the impact. Based on what we see the backlog at on Monday or Tuesday we might send the email to 10-20% of current ARP1 participants just to give a little bit more of a nudge to speed up the project and see if we can get the backlog down to around 500 - 1000 results ready to send. Our goal is to have steady supply and minimal backlog and we are almost there.

Kevin, Please post you findings on Monday or Tuesday. If needed, I can throw about 200 threads at the problem.

I didn't wait. I already moved 80 cores over and another 128 coming shortly
Problem is, I seem to be getting mostly gens 78, 79, and 80 which tells me the older generations are already out in the wild since they have priority. Adding more cores looks like it might actually extend the range between the front and back generation.