Re: trickles

Expanding on SekeRob's post above, trickles enable work on a single trajectory (30+ WUs) to continue as quickly as possible. If a WU trickles at 50k steps and then disappears, we send out another WU to a different volunteer picking up at 50k steps. While some complete trajectories never experience this problem, the vast majority of the thousands that have already run have had to be restarted from a trickle message, often several times, so we do see a benefit from them on our end. But I understand the pushback against trickles. If the AsyncRE WUs are short enough, then we may not need to rely on trickles to ensure trajectories aren't stalling. I'm fairly certain reliability scores per host are already baked into the BOINC scheduler, but I don't know if a dichotomous "reliable hosts don't have to trickle" and "unreliable hosts do" scheme is feasible (or necessarily desirable).

Re: invalid trickles

Not sure, will ask around.

Re: AsyncRE manager


Uplinger (and others) are integrating a BOINC-native implementation on the BOINC head server with the intent to make the exchange part of replica exchange ("information sharing") be as quick and efficient as possible with little additional CPU load. Luckily we won't be exchanging information between every replica currently running, only among groups which correspond to the same protein-ligand complex, which cuts down on the CPU cycles needed for the exchange process.


We currently face the same problem when the validator gets stuck since new WUs rely on the output previous WUs. AsyncRE essentially just changes how to choose the parameters assigned to each new WU (but these choices are key in accelerating the simulations properly).

Think you can read back in a Uplinger post when the trickle validator last stalled for longer, it took several hours to catch up. It's like a whole extra results set per day, effectively all combined the scheduler thus seems to handle over 2 million 'validation attempt' per day, in the present setup.

With that much WU failure happening, it can't just be attributable to volunteers aborting jobs or turning off their computers, etc. - it sounds much more like lots of WUs are being sent out with major birth defects. In other words, the project is royally screwing up WU production and the trickled-upon volunteers are paying the price! Is that correct?

@Doneske re: AsyncRE bottlenecks, etc.

In terms of current WUs from FA@H2, there already exists a background process which builds the WUs by parsing parameters from a configuration file and parsing the step number from the last output corresponding to that trajectory. AsyncRE will parse a bit more (certain energy values, etc.) reported in the last output file, but otherwise only is adding on the exchanging mechanisms. While the Python implementation of AsyncRE available on Github has lots modules to support different architectures, the WCG version be stripped down to only include the essentials. So in terms of *additional* CPU load to the head server, that should only come from the exchange routine, which we hope will be minimal. (I can't really comment on the current system hitting hardware limits since I can only see that from my own BOINC logs.)

In terms of GPGPUs, while AsyncRE (the framework for asynchronous replica exchange) supports GPUs, the MD engine IMPACT currently does not have a BEDAM GPU implementation. So for now GPGPU computing for this project isn't on the table (though collaborators at Brooklyn College are working on the necessary components to make this a possibility).

@Papa3 re: bad workunits

Bad WUs tend to blow up pretty quickly. If you get to the first trickle without problem, the workunit is most likely fine. So I would attribute the "restarted-by-trickle" WUs to be a result of volunteers aborting/restarting/logging off.

Anything that will improve the accuracy and speed of screening for potential therapeutic drugs for any disease will be a great advance.
Thanks for your hard work, Team Levy.

However. there is something about running ARE on WCG that I haven't seen mentioned. As I understand ARE, in order to be able to exchange info between WUs running on different members' machines, the WUs would need to be running concurrently. That may be simple to arrange on an in-house cluster, but us members have different work cache and project mix settings, so some replicas will want to be finished before others have even started. If my understanding is correct, how do you propose to overcome this?