| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Completed Research
Forum: Computing for Clean Water Forum
Thread: High writes to disk for C4CW |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 15
|
|
| Author |
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges: 
|
I am setting up C4CW on a Win7 64-bit machine, with a dedicated i5-3550 CPU (3.5 GHz) having all four cores devoted to C4CW.
Somewhat surprisingly, according to Task Manager the writes to disk are in the range of about 170 to 200 GB/day. That is the range that I normally associate with CEP2, but have not seen that much on another project before. It is no big deal for me; I will set up a ramdisk (Primo Ramdisk) in order to protect my SSD, but have not seen this mentioned before. |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
I've never noticed and running since a week 7 concurrent on my octo, the 8th core exclusive for CEP2. I've though upped the default Write to disk "at most" for years to at least 5 minutes [SSD owners would want to max this value way up]. Thw WtD has no effect on CEP2, but other sciences that frequently checkpoint do, which cuts the traffic big-time.
BTW, There's softwrites... they never go to disk. The value of interest is the write delta and page faults delta. Just fired up Sysinternals' Process Explorer, selected a C4CW task, then properties and the Disk/Network tab and let it sit for a while... no traffic to mention, also the Performance tab showing nothing at all. Got enough memory allocated so there's no incessant memory to swap file exchanging? |
||
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges:
|
I've never noticed and running since a week 7 concurrent on my octo, the 8th core exclusive for CEP2. I've though upped the default Write to disk "at most" for years to at least 5 minutes [SSD owners would want to max this value way up]. Thw WtD has no effect on CEP2, but other sciences that frequently checkpoint do, which cuts the traffic big-time. Good point. I had never paid any attention to this setting.BTW, There's softwrites... they never go to disk. The value of interest is the write delta and page faults delta. Just fired up Sysinternals' Process Explorer, selected a C4CW task, then properties and the Disk/Network tab and let it sit for a while... no traffic to mention, also the Performance tab showing nothing at all. Got enough memory allocated so there's no incessant memory to swap file exchanging? I will try it the next time I write to disk. But with a ramdisk, I don't think it shows up in Process Explorer as page faults in Virtual Memory or writes in the I/O section, since it is not going to the disk. At least on Task Manager it doesn't, since it is not an actual write to the disk drive anymore. But I have 8 GB memory, and am now using 5 GB for the ramdisk, which is more than enough (I made it purposely larger than necessary, since I might want to do CEP2 on this machine later). That still leaves 3 GB working memory, while it is only using about 800 MB. |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Still I do not understand how you get 170-200GB day writing with 4 threads, when I see next to nothing for 7 C4CW threads and only regular spinning 500GB HD, 8GB RAM, just observing 112Kb for a checkpoint and 32Bytes as a delta.
Been reading lots about how to cut the SSD writing and optimize. Would that be that much for just 4 threads, wonder what WCG will be looking at when just reading they want to put part of the server storage on SSD systems and their 158GB live storage DB. |
||
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges:
|
Still I do not understand how you get 170-200GB day writing with 4 threads, when I see next to nothing for 7 C4CW threads and only regular spinning 500GB HD, 8GB RAM, just observing 112Kb for a checkpoint and 32Bytes as a delta. I don't know why the difference either, but did you look at Task Manager/Resource Monitor/Disk Activity? It shows only the background processes for me now, since I have put the BOINC data folder on ramdisk. Been reading lots about how to cut the SSD writing and optimize. Would that be that much for just 4 threads, wonder what WCG will be looking at when just reading they want to put part of the server storage on SSD systems and their 158GB live storage DB. They might want to investigate a caching program in that case. I have used FancyCache with good results. It is currently in beta (free 180 day trial). http://www.romexsoftware.com/en-us/fancy-cache/index.html It can cache both reads and writes, but the reads are usually not much of a problem for an SSD. However, using a write cache with a long write delay (24 hours), it cut down the writes on CEP2 by two orders of magnitude for me (i.e., only 1% got written to the disk). I ultimately decided on a ramdisk, since everything I need to cache is conveniently in the BOINC data folder, and it eliminates the writes entirely for that, but for a database the caching software might be better. In theory, the ramdisk may be more stable, since it does not try to cache system functions, but I did not have stability problems with FancyCache, at least in the later betas. You need a good backup power supply (UPS) with automatic shutdown in either case, since you will lose data if the power goes out, or the machine crashes. Fortunately, my machines are stable. [Edit 1 times, last edit by Jim1348 at Jan 10, 2013 10:19:20 PM] |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
The only process I see in TM Resource Manager, Overview Disk section, sorting by process name is BOINC.exe at 410 byte/sec. In The Disk tab I don't see not even that, only Firefox and flipping back to the overview boinc.exe is gone, so it probably was doing some scheduling action. Maybe a screenshot would help us get on the same page, for next if you revert to running BOINC off your SSD.
|
||
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges:
|
OK, I switched back to the SSD, and this is typical of what I see, though it bounces around a lot. (I have not changed the checkpoint frequency, so that could be reduced somewhat.)
 |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Yes, found it before you posted and was watching the HD light which blips briefly every few seconds, but just briefly. Don't think that what we see is actually hitting on the disk. Picture you copying 200GB from one disk to another and consider if you would notice. I think so. Did so yesterday, 98GB from the system disk to a USB 3.0 external drive [at a rate of 49.5MB per second] That ran non-stop for 30 minutes or so.
|
||
|
|
Jim1348
Veteran Cruncher USA Joined: Jul 13, 2009 Post Count: 1066 Status: Offline Project Badges:
|
Yes, found it before you posted and was watching the HD light which blips briefly every few seconds, but just briefly. Don't think that what we see is actually hitting on the disk. Picture you copying 200GB from one disk to another and consider if you would notice. I think so. Did so yesterday, 98GB from the system disk to a USB 3.0 external drive [at a rate of 49.5MB per second] That ran non-stop for 30 minutes or so. There is a write-to-disk measurement in FancyCache also (to show how much writing to the disk that it saves), and as I remember the results are comparable to what I see in Task Manager. I don't think it endangers the SSD so much for Clean Water as it does for CEP2, which is even higher, but I will play it safe. |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
I also had this issue with BOINC in general. I managed to write 2.2TB (terabytes) before I noticed BOINC was eating my SSD alive. I wouldn't mind this on a mechanical hard drive, but I'd like to avoid stress-testing a new Samsung SSD for a while.
The only way I managed to control my writes while running BOINC was by disabling or lowering virtual memory on my system SSD and installing the BOINC data directory to a different hard drive (you can still install the program to your main hard drive). For those that don't know how -- you go to the Control Panel-->System-->Advanced System Settings-->Advanced Tab-->Settings-->Advanced Tab-->Change under virtual memory. I only have 8GB of RAM, so this might be part of the reason why (Windows uses virtual memory even though you still have plenty of available memory -- available memory being comprised of standby and free memory). I don't consider this a solution, but it's worth a try if you're still experiencing high writes to disk. You're also more susceptible to data loss, but it shouldn't be a problem if your system is stable. Checkpoints still occur at regular intervals. Hope it helps. |
||
|
|
|