| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Active Research
Forum: OpenPandemics - COVID-19 Project
Thread: OPN1 WU memory leaks? |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 41
|
|
| Author |
|
|
[CSF] Aleksey Belkov
Cruncher Russian Federation Joined: Feb 28, 2013 Post Count: 3 Status: Offline Project Badges: 
|
Good day
----------------------------------------Today I noticed Memory Exhaustion on my main workstation due almost all opn1 WUs used much more Virtual Memory then usual(up to 2.5 GiB vs 200-250 MiB per WU).  After restarting BOINC client/manager on this host, Virtual Memory usage by opn1 WUs normalized(200-250 MiB per WU). But later I noticed an increase in the consumption of virtual memory by opn1 WUs again. And the growth of consumption continues! See posts below Any thoughts what could be the cause of such a problem? Tech Info: Main host: AMD Rysen Threadripper 2950X (c16/th32) / 64 GB ECC RAM / OS: Windows 10 Enterprise LTSC x64 Other(7) hosts: Intel Core i5 3470 (c4/th4) / 8-16 GB RAM / OS: Windows 10 Enterprise LTSC x64 [Edit 3 times, last edit by [CSF] Aleksey Belkov at Oct 6, 2021 1:47:51 AM] |
||
|
|
Mike.Gibson
Ace Cruncher England Joined: Aug 23, 2007 Post Count: 12436 Status: Offline Project Badges: 
|
Aleksey
I cannot answer your question, but we have suddenly seen the batch numbers going up by thousands with very small numbers in the batches, as posted in Work Avaiable. There may be a connection. Mike |
||
|
|
Crystal Pellet
Veteran Cruncher Joined: May 21, 2008 Post Count: 1323 Status: Offline Project Badges: 
|
I've seen the same with these tasks with memory extend
OPN1_0064414_00035_0 613.88 MB 492.34 MB from 2.6GB at first OPN1_0063979_00005_0 2611.86 MB 2501.57 MB OPN1_0064151_00026_1 295.47 MB 721.60 MB from 2.5 GB at first OPN1_0064373_00024_1 Computation error (29539,) Memory reduced after suspend task (LAIM off) and resume. |
||
|
|
[CSF] Aleksey Belkov
Cruncher Russian Federation Joined: Feb 28, 2013 Post Count: 3 Status: Offline Project Badges:
|
Mike,
Thank you for your comment. It would be great somehow draw the attention of the project engineers to this problem. |
||
|
|
[CSF] Aleksey Belkov
Cruncher Russian Federation Joined: Feb 28, 2013 Post Count: 3 Status: Offline Project Badges:
|
Crystal Pellet
----------------------------------------Thanks for feedback. Now I know that this is not a local host-specific problem. Now I see that the problem has started to reveal itself on other hosts in my crunch-pool. To rule out possible problems with memory exhaustion(as a workaround), I decide to temporally exclude OpenPandemics WUs from receiving and abort all cached opn1 WUs. [Edit 1 times, last edit by [CSF] Aleksey Belkov at Oct 4, 2021 10:05:35 PM] |
||
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7697 Status: Offline Project Badges: 
|
That runaway memory problem has hit me also. So far it has only affected Linux hosts and not the one Windows machine I am running. I have suspended all of the running OPN programs and will restart them one at a time to try to nurse them to completion. I will switch my allocation to MCM for the time being until there is a fix for the OPN problem.
----------------------------------------On a 32 thread system with 16gb of memory the sysmon showed 15.9gb of memory in use plus a ton of swap space. This system usually runs under 4 gb of memory in use. The system was sluggish as a drunk snail until I got a bunch of the jobs suspended. Edit:spelling Cheers
Sgt. Joe
----------------------------------------*Minnesota Crunchers* [Edit 2 times, last edit by Sgt.Joe at Oct 5, 2021 12:46:20 AM] |
||
|
|
Grumpy Swede
Master Cruncher Svíþjóð Joined: Apr 10, 2020 Post Count: 2209 Status: Offline Project Badges: 
|
I'll abort all my OPN1 tasks from my Low memory Laptop, and start with 100% MCM1 tasks instead.. I hope the project staff takes care of this problem soon.
----------------------------------------Edit: Aborted them from my 16GB computer also. Goodnight for now, OPN1. [Edit 2 times, last edit by Grumpy Swede at Oct 5, 2021 1:04:46 AM] |
||
|
|
pvh513
Senior Cruncher Joined: Feb 26, 2011 Post Count: 260 Status: Offline Project Badges:
|
Not sure if it due to a memory leak, but the memory consumption has gone up on my linux machines. I now frequently run into oom-kill due to the machine being out of memory (I typically have 1 GiB per thread). IIRC the memory consumption of OPN used to be quite modest.
----------------------------------------Edit: I have now also aborted all OPN tasks, they were simply putting too much strain on my computers. [Edit 1 times, last edit by pvh513 at Oct 5, 2021 6:12:59 AM] |
||
|
|
wujj123456
Cruncher Joined: Jun 9, 2010 Post Count: 38 Status: Offline Project Badges:
|
Same here. Many of my OPN1 WUs are now exceeding memory usage of ARP1, which I had special config for. It would be good to know if it's a memleak bug, or it's intended behavior change. The latter would likely make me apply similar limits to ensure I can fill all the cores without being limited by memory.
|
||
|
|
Crystal Pellet
Veteran Cruncher Joined: May 21, 2008 Post Count: 1323 Status: Offline Project Badges:
|
To add to my previous post:
----------------------------------------My high memory OPN's were on a Windows 10 laptop with 8GB RAM and 4 threads running. The result of the task that crashed because of 'out of memory': https://www.worldcommunitygrid.org/contribution/results/1951023765/log *** Dump of the Process Statistics: *** Normally the CPU-OPN's only do 1 or 2 jobs with max 50 compounds, but the task that crashed was busy with job 21. Edit: Added extract of log [Edit 1 times, last edit by Crystal Pellet at Oct 5, 2021 7:01:50 AM] |
||
|
|
|