Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 76
Posts: 76   Pages: 8   [ Previous Page | 1 2 3 4 5 6 7 8 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 86369 times and has 75 replies Next Thread
Speedy51
Veteran Cruncher
New Zealand
Joined: Nov 4, 2005
Post Count: 1220
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: RAMDisk

Thanks for help . When I run CEP2 I'll run with a HCMD etc. Thanks again
----------------------------------------

[Nov 15, 2010 2:37:27 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Coleslaw
Veteran Cruncher
USA
Joined: Mar 29, 2007
Post Count: 1343
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: RAMDisk

How well would a Ram Drive work in a Virtual Machine environment? If you have 12GB ram and give say 3GB to 2 virtual machines each and then used the other 6GB for the 2 cores not dedicated to the Virtual Machines....would a Quad Core run better that way? That would allow you to extend the Ram Disk in essence to greater then 3.5 and still stay free because you could make 3 on the same machine. I don't have the extended knowledge and number crunching abilities to come up with an efficiency or worthiness rating, but that would be one solution to having more then just 3.5GB for ram disk usage.

Edit: To clarify, I meant putting a ram disk in each of the virtual machine environments for a total of 3 over all on the single quad core scenario.
----------------------------------------

----------------------------------------
[Edit 1 times, last edit by Coleslaw at Nov 15, 2010 8:22:28 AM]
[Nov 15, 2010 8:21:07 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: RAMDisk

im seeing some extremely nuts performance using fdupes to knock down the IO load. see nieghboring thread. http://www.worldcommunitygrid.org/forums/wcg/viewthread_thread,30419

i think we have been attempting the same goals.
[Nov 28, 2010 10:20:36 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Jim1348
Veteran Cruncher
USA
Joined: Jul 13, 2009
Post Count: 1066
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: RAMDisk

I am getting 97+ percent CPU efficiency on my quad-core when using FancyCache (write-cache only is sufficient), with 3 CEP2 jobs and a HPF2 job running. Before using the cache, I could only get about 96% on a single CEP2 job, due to the I/O limitations of my Gen 1 SSD, and it would go down to about 89 to 92 percent with 4 jobs running. I expect on a spinning platter drive it would reduce thrashing, but I have not tried it there. I have found that the "Disks" version of FancyCache is more stable than the "Volumes" version, but YMMV.
[Nov 28, 2010 2:03:12 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Bearcat
Master Cruncher
USA
Joined: Jan 6, 2007
Post Count: 2803
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: RAMDisk

Would having a 2nd disk strictly for the write cache work? If so, how do tell boinc to do this in ubuntu?
----------------------------------------
Crunching for humanity since 2007!

[Nov 29, 2010 12:23:51 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: RAMDisk

static uncompressed files are essentially part of a rolling ram disk in linux.

everything in linux that lives on a block device has a cache lifespan unless reads / writes are done expressly as raw.

so if you have more physical ram than disc, you are almost assured that you'r drive will be cached unless you have some really bizarre memory usage in a program.

ways to improve under linux:
use fdupes to hardlink -- my own experiment

on a platter you will get good results from running inside of a perhaps 8-gig contiguous extent loopback btrfs filesystem with compress option, without needing to do any serious OS reinstalling.

i think this would be something like




somewhere=/tmp
GIGS=8
cd $somewhere
sudo bash -li
dd if=/dev/zero of=qextent.btrfs bs=128M count=$[ ${GIGS} * 16 ]
mkfs.btrfs qextent.btrfs
mkdir t
mount -oloop,compress,noatime qextent.btrfs /tmp/t -tbtrfs
#stop boinc here
rsync -ahP /var/lib/boinc-client/. t/
chown -R boinc:boinc t/
mount --move t /var/lib/boinc-client
#restart boinc here


this volume remains in qextent.btrfs after reboot but if you use /tmp it WILL BE SWEPT after reboot

to use this setup after reboot/unmount


mount -oloop,compress,noatime $somewhere/qextent.btrfs /var/lib/boinc-client


before boinc begins.

alternatively similar can be done on your swapfile by mounting ramzswap or tmpfs and running the queue inside swap or compressed swap., often with machines in a colo there's little reason NOT to do experiements on gigantic swapfs volumes :)

being on a small extent of disc obivous packs the data together on a less pathological range of fragmentation hardware access seeks.

swapfs is superior to other write overhead because there's no accounting or journaling, at the cost of RAM to other processes if you thrash it.

ramzswap is an option i have been curious about however you can have the same basic thing from btrfs with -ocompress so i have kept it simple.

i haven't tested a loopback mount arrangement yet, but on an ssd there's no point really.
----------------------------------------
[Edit 4 times, last edit by Former Member at Nov 29, 2010 7:54:33 AM]
[Nov 29, 2010 3:22:09 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Bearcat
Master Cruncher
USA
Joined: Jan 6, 2007
Post Count: 2803
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: RAMDisk

So, in ubuntu, is more ram better or create a ram disk or use another program to help? I have my 8 core harpertown PC with ubuntu 64 dedicated to this project at this time. I recently added clean water to help reduce cep2 from running on all 8 cores simultaneously. I currently have 8 GB memory installed but if it would benefit to have more, then would do so. Which is more efficient?
----------------------------------------
Crunching for humanity since 2007!

[Nov 29, 2010 6:41:53 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: RAMDisk

Bearcat ubuntu or any other linux with 64 bits is as efficient as it gets standalone before tweaking advanced stuff.

so for the advanced stuff:

i have been crunching for maybe 2 weeks but

a) to improve seek times and match up your ram size to disc patterns: if you're not on a SSD and if you have 8 gigs of ram try running 8 gigs of max filfesystem space for your boinc-client as outlined above -- for 8 gigs of RAM create an 8 gig file with "dd", then move your queue data into it

if you can't squeeze all your BOINC data into the 8 gig loopback filesystem, be it by copmression or size alone, you can't roll 100% in RAM either.

b) to merge redundant queue files concurrently on fc12+ or ubu 10.10: with or without the loopback filesystem or running in tmpfs you can try to do it with fdupes. this WILL help with an SSD as well. (version of fdupes matters, see other thread https://secure.worldcommunitygrid.org/forums/...ead,30419_offset,0#304927 )

(paste into bash shell )

{ while fdupes -Lrnq /var/lib/boinc-client ; do sleep 600;done  } >/dev/null  & disown 

i dont have statistics or even know reliable metrics to measure before or after results but if you can run fdupes in a short amount of time (8 gigs oughta do it fine) you should be able to reduce IO waits between code running in the cpu.
.
----------------------------------------
[Edit 2 times, last edit by Former Member at Nov 29, 2010 7:51:31 AM]
[Nov 29, 2010 7:40:00 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Jim1348
Veteran Cruncher
USA
Joined: Jul 13, 2009
Post Count: 1066
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: RAMDisk

I am now using the licensed version of Dataram RAMDisk (only $10 USD) with 8 GB of DRAM running under Win7 64-bit on a quad-core. By carefully choosing the size of the RAMDisk (5.8 GB, which shows up as 5.65 GiB as installed), I am able to get it to run CEP2 on all four cores without problems thus far. It leaves just enough for the OS in the remaining memory, though it does bump some things over to the page file, but that is no problem. I think it probably only allocates space for the page file without doing much with it anyway. But my actual RAMDisk memory usage is much less, and it would happily run in the 4 GB space of the free version of Dataram if only it would download the new CEP2 projects when the old ones complete. (My figures are the same as what KerSamson gets most of the time,
https://secure.worldcommunitygrid.org/forums/...ad,30422_offset,60#306991 though it does get near to 4 GB occasionally, but not over that thus far).

I know there was some beta testing where a smaller space requirement caused problems, but I am wondering if that question couldn't be revisited, or maybe it could be made an optional parameter like some of the other requirements? Probably a lot more people could get the benefits of a RAMDisk and solve a lot of the problems we are seeing if that were possible. The big users run their machines 24/7 with backup power supplies anyway, and so the risk of data loss is minimal, though it should be considered.
----------------------------------------
[Edit 3 times, last edit by Jim1348 at Dec 15, 2010 1:27:24 PM]
[Dec 15, 2010 1:15:28 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: RAMDisk

applause
[Dec 15, 2010 2:30:13 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 76   Pages: 8   [ Previous Page | 1 2 3 4 5 6 7 8 | Next Page ]
[ Jump to Last Post ]
Post new Thread