This machine is a dual core Pentium E5800 running off a 16 GB USB flash drive. I think the computer only has USB 2.0, while the drive is USB 3.0.

Both of the work units were at the 99% mark and doing their final compression and checkpointing, and they both error'd out at the last second again. They were both at the ~36 hour mark and pretty much complete!



Looking at the Event Log:


I think with both ARP1 work units doing their final compression of output files or whatever at the same time, there is a LOT of disk activity, and since this USB3 flash drive runs at USB2, it's even slower. I think the reason for these errors is BOINC loses patience with the disk I/O and thinks that files no longer exist, when the disk is simply busy working.

Sucks that I lost two complete work units at 36 hours each. I could potentially put an SSD or even an older spinning HDD into this computer, but it's not worth either the cost or effort. I dunno, as most ARP1 tasks complete without error.

Just a thought here too. It just be the coincidence of the units finishing in close proximity to each other, but I don't really know. What I do know is I have two 24 thread systems and an 8 thread system all running off of 16gb flash drives. I limit each of the systems to 4 ARP units, but I have never seen more than 3 on any one system at a time. They have all functioned so far without any errors on the ARP units. They are all USB 2 units. They may eventually wear out, but they have been going a couple of years already. I suspect it is not the hardware, but it certainly could be if the USB units might be failing.
Cheers

This 'segmentation violation' makes me think of this "Segmentation violation" thread. When the error occurs, the ARP1 program exits immediately, meaning that the compression algorithm also stops at that point, which means that the compression results are unusable, they are considered 'lost', so the final output files can no longer be written and they will be empty.

Here is what you will normally see in the program's slot directory in the file 'stderr.txt':

Starting WRFMain
[01:41:46] INFO: Checkpoint taken at 2018-07-03_06:00:00
[04:08:20] INFO: Checkpoint taken at 2018-07-03_12:00:00
[06:35:09] INFO: Checkpoint taken at 2018-07-03_18:00:00
[08:20:03] INFO: Checkpoint taken at 2018-07-04_00:00:00
[10:22:31] INFO: Checkpoint taken at 2018-07-04_06:00:00
[12:56:38] INFO: Checkpoint taken at 2018-07-04_12:00:00
[15:14:59] INFO: Checkpoint taken at 2018-07-04_18:00:00
[16:57:53] INFO: Checkpoint taken at 2018-07-05_00:00:00
INFO: Simulation complete compressing output.
16:59:49 (24943): called boinc_finish(0)

I think this is what happens normally, in this order:
ARP1 algorithm runs
ARP1 algorithm finishes (after 8 checkpoints: 'Simulation complete')
ARP1's compression algorithm starts
ARP1's compression algorithm finishes
ARP1 output files are written
ARP1 result files are uploaded

As soon as anything goes wrong during one of these steps, especially the first three are crucial here, then there is nothing to upload, except empty output files.

ARP1 uses a lot of memory. After the error occurred twice on my machine, I decided to install more memory (RAM), effectively doubling the amount. Last night I had 8 ARP1s running at the same time on my machine without a hiccup. (Received 5 ARP1s yesterday at the same time (06:58:29), another one at 13:19:58 and two more at 21:24:48. While the first 5 of them have already finished, the last three are still running now.)

Result Name          Status                 Sent Time     Due / Return Time  CPUh/Spent Claimed/Granted
ARP1_0016068_001_0-- Pending Validation 12/19/19 05:58:29 12/20/19 05:49:55 23.16/23.76   971.2/0.0
ARP1_0010317_001_1-- Pending Validation 12/19/19 05:58:29 12/20/19 05:47:05 23.18/23.78   972.1/0.0
ARP1_0015077_001_1-- Pending Validation 12/19/19 05:58:29 12/20/19 04:53:18 22.30/22.85   934.2/0.0
ARP1_0015073_001_0-- Pending Validation 12/19/19 05:58:29 12/20/19 04:39:43 21.96/22.50   991.8/0.0
ARP1_0008207_001_0-- Valid              12/19/19 05:58:29 12/20/19 04:37:40 21.98/22.52 1,008.6/950.0

[Generated by wcgformat]

Almost never. These machines are running Linux. The only time they get rebooted is when I have a power outage, which is not often.
Cheers