Might have something to do with these new WUs that have significantly more jobs inside them. You nay have seen runtimes increase 3-6x longer than before.

Not sure how many concurrent tasks you are running, but might that be crashing your GPU now with these new WUs, thus causing the Error?

Either way, these new WUs just suck!

I've had a quick look through the sources, and I can confirm that we're both right - 255 doesn't occur in BOINC, either as an exit code or an error number.

I've also looked at the autodock-vina sources on GitHub, and it doesn't appear in there either. There are plenty of abnormal end-cases, but they all print a text description and exit with EXIT_FAILURE. That isn't defined locally, so it must be the standard language code, which would be 8. Not 255.

So that leaves us with the actual executable program being run, which is "wcgrid_opng_autodockgpu", with various version/platform/device specifiers on the end. I don't think the specific wcgrid modifications to the underlying autodock code are open source? Autodock itself is under an Apache open source licence, but there's no separate licence file in the derivative version distributed here (and I don't think Apache requires one). We might need input from an IBM or Krembil tecchie on that.

The tasks all appear to run correctly for some time until they quit. Then some other task for another project takes its place. No gpu has ever fallen off the bus and they all continue to run normally with other projects work.

I have 32 GBs of RAM each in three hosts and 128 GB of RAM each in two other hosts.
There shouldn't be any RAM pressure even with these supposed bigger work units.
I have enough to run the python task from GPUGrid easiliy.

So without any clue as to what is wrong with all of my hosts ONLY with WCG, I have suspended the project. No point in wasting bandwidth or time with these tasks that all fail.

I'm with Richard on this one! I reckon it's an issue somewhere in the wrapper code (either checkpoint management or the set-up/tear-down of individual GPU sessions...) And my reasoning? As mentioned further up-thread , these tasks are much larger in terms of the amount of output produced, and in the intensity of said output. On one of my systems I can actually see the effects of this on overall BOINC agent behaviour! Thus...

I only have two systems which run GPU tasks - both are on Ubuntu 20.04 and have reasonably up-to-date approved NVIDIA drivers (470.86 at present). On both systems there are other things going on, including monitoring the arrival and completion of BOINC tasks(!), so I don't let BOINC have too many cores...

My i7-7700+GTX1050Ti has a usual work mix of two ARP1, two MCM1, one OPN1 and a non-WCG GPU job; it just drops one of the WCG CPU tasks them handles these without seeming in the least bit concerned (though disk activity does increase somewhat...).

My Ryzen 3700X+GTX1660Ti has a usual work mix of 3 ARP1, five MCM1, one OPN1 and a non-WCG GPU task will try to run two OPNG at once if it has them. I've got one spare core allocated to WCG to allow for the first one, and the second one kicks out a CPU task... And that's when the fun starts! -- once I have two OPNG tasks running, the disk activity light comes on and stays on!!! And if I look at disk I/O stats it isn't kidding! It seems to be the same whether it has kicked out an ARP1 task or not -- ARP1 is a bit wordy too, but nothing like OPN1/OPNG on short jobs!

Now you might be thinking "So what? It does a lot of I/O!" But it definitely chokes up BOINC task wrap-up; my monitoring scripts allow about 90 seconds from the completion entry appearing in the project's job log to search for the stderr section in the results block for the task in client_state.xml) and there are occasions where it gives up and [if I notice in time] I have to get the data off the web site instead. The worst case of this that I've seen (during the OPNG soak test in 2021) actually resulted in a "Finish file not found" message using a recent client! (DIdn't they stretch the time for that to something like 5 or 10 minutes?) It only seems to have these problems when OPNG is running (or when I used to have lots of OPN1 running - same bulk I/O volumes?...) I don't know how much influence the storage medium might have on the sluggishness - I'm not using SSDs for BOINC storage...

Now, if there are any watchdog threads associated with OPNG and something holds up progress for too long, maybe the entire process gets killed without benefit of any error messages! This is speculative, based on my experience of sluggish BOINC agent work with two OPNG tasks, and the fact that lots of folks with lesser equipment (and less time stress per job) don't seem to be having problems at all (and are saying so!), whilst I suspect that the people being bitten have systems most likely to overtax the wrapper/agent code (e.g. multiple GPUs, lots of tasks running at once. Keith's case in point had a powerful GPU - I wonder how many tasks that user is running at once?!?)

Anyway, without access to the WCG OPNG-specific BOINC interface and task management code, we are left guessing (and I suspect that unless they start getting workunits lost because no-one can run them, there won't be any effort put into sorting it out at the moment)

I notice that Keith has posted a plea for help in the GPU Support Forum. Worth a try!!! In another, older, thread in there, adriverhoef has recently posted about a different issue with these long jobs, and I remarked that I've tried to use "Contact us" to ask for them to put a ceiling on the number of jobs in a task (as they did for OPN1!) -- I have no idea whether contact was successful...

So now we wait... :-(

Cheers - Al.

P.S. PMH-UK is right about the 8-bit 255 error code (though well-written code should be able to cope with memory management issues and at least tail gracefully! It's [also] a generic O/S response for "What the heck happened" cases, unrecognised error states, et cetera, and could well be a result of a forced termination not done via a signal...

[Edit - added clarification to when I lose task monitoring output...]

Alan, on the question of how loaded my systems are . . . . all of them are running at 90% or less. I always keep 2-4 cores free from BOINC use for system maintenance activity.

I never have any sluggishness noted in the UI on any of my hosts.

Storage is by M.2 PCIE SSD with none approaching 25% of storage space allocated.

I too notice the OPNG tasks hitting the storage activity LED more actively than previous tasks.