Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 
World Community Grid Forums
  Category: Active Research
  Forum: Africa Rainfall Project
  Thread: BIG file size! Hope you have a lot of bandwidth available

Quick Go ยป
No member browsing this thread
Thread Status: Active
Total posts in this thread: 49
Posts: 49   Pages: 5   [ Previous Page | 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 45854 times and has 48 replies Next Thread
Jack007
Master Cruncher
CANADA
Joined: Feb 25, 2005
Post Count: 1604
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
So yeah memory doesn't seem anywhere near one gig, but when I only ran 2 Africa units (no other units at all) they were all less than 11 hours. Now that i'm running 4 Africa and 10 MIP, Africa looks like it will be around 15 hours.
----------------------------------------
[Nov 4, 2019 2:44:08 AM]   Link   Report threatening or abusive post: please login first  Go to top 
alanb1951
Veteran Cruncher
Joined: Jan 20, 2006
Post Count: 982
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
So yeah memory doesn't seem anywhere near one gig, but when I only ran 2 Africa units (no other units at all) they were all less than 11 hours. Now that i'm running 4 Africa and 10 MIP, Africa looks like it will be around 15 hours.

Yup - MIP is a real performance killer! You may or may not be aware that it thrashes the L3 cache, which will have a knock-on effect for other applications as you are observing.

The rule-of-thumb for MIP1, cited more than once in these forums, would seem to be not to run more than 1 MIP1 task per 4MB of L3 cache! Have a look at how long your MIP1 tasks are taking, and try only running (say) 2 at a time for a while and see how long they take then; your ARP tasks will also thank you!

(MCM, and SCC when available again, constitutes a good use of spare cores as such work places almost no stress at all on L3 cache!)

Good luck finding a workload mix that suits you!

Cheers - Al.
[Nov 4, 2019 4:11:39 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
So yeah memory doesn't seem anywhere near one gig, but when I only ran 2 Africa units (no other units at all) they were all less than 11 hours. Now that i'm running 4 Africa and 10 MIP, Africa looks like it will be around 15 hours.
Whatever you can read from the task state file after 8 hours running and 2 checkpoints
<active_task>
    <project_master_url>http://www.worldcommunitygrid.org/</project_master_url>
    <result_name>ARP1_0004399_000_1</result_name>
    <checkpoint_cpu_time>24258.550000</checkpoint_cpu_time>
    <checkpoint_elapsed_time>26678.448229</checkpoint_elapsed_time>
    <fraction_done>0.250000</fraction_done>
    <peak_working_set_size>805240832</peak_working_set_size>
    <peak_swap_size>1061302272</peak_swap_size>
    <peak_disk_usage>657681929</peak_disk_usage>
</active_task>
[Nov 4, 2019 9:43:32 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Jean-David Beyer
Senior Cruncher
USA
Joined: Oct 2, 2007
Post Count: 339
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
Yes climate apps have a lot of data involved as those of us who regularly crunch for Climate prediction.net know. Still haven't managed to get any of these but at least they are not like one of the recent beta apps for CPDN where uploads were over 1GB!


Am I the only one with wide-band Internet access? I presumably have 75 Megabits/second up and down; if only the servers had the same! That is 9.38 MBytes/sec. But sometimes I even get more.
Date 	Download 	Upload 	 	Test Server 
11/13/2019 2:20:58 PM	82.91 Mbps	43.92 Mbps	New York City, NY	 	  	
11/9/2019 9:32:03 AM	82.44 Mbps	36.14 Mbps	New York City, NY	 	  	
10/4/2019 7:45:44 AM	83.98 Mbps	87.33 Mbps	New York City, NY


At 9.38 MBytes/sec, 1 GByte would take 107 seconds, or less than two minutes.
----------------------------------------
[Nov 13, 2019 7:31:10 PM]   Link   Report threatening or abusive post: please login first  Go to top 
DrMason
Senior Cruncher
Joined: Mar 16, 2007
Post Count: 153
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
Hey Jean-David, I'd say you're one of the lucky few, at least as far as I've experienced. The only reasonable broadband ISP in my area is massively asynchronous. Download is ok, but my upload is 20x slower.

As far as I know, most cable internet systems are set up to be asynchronous. But synchronicity is becoming more common because of fiber, so I'm hoping that eventually my area gets upgraded! It hasn't struck me as a massive inconvenience, because I knew what to expect going in. The people I really feel for are the ones on dial-up or on satellite networks. They might have to give it some thought before contributing...
----------------------------------------
[Nov 14, 2019 1:26:39 AM]   Link   Report threatening or abusive post: please login first  Go to top 
retsof
Former Community Advisor
USA
Joined: Jul 31, 2005
Post Count: 6824
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
Hey Jean-David, I'd say you're one of the lucky few, at least as far as I've experienced. The only reasonable broadband ISP in my area is massively asynchronous. Download is ok, but my upload is 20x slower.

As far as I know, most cable internet systems are set up to be asynchronous. But synchronicity is becoming more common because of fiber, so I'm hoping that eventually my area gets upgraded! It hasn't struck me as a massive inconvenience, because I knew what to expect going in. The people I really feel for are the ones on dial-up or on satellite networks. They might have to give it some thought before contributing...

A farm friend is on a satellite network. Whenever a gray cloud with water goes over and it can be a small one, it takes 15 minutes to come back up again.
----------------------------------------
SUPPORT ADVISOR
Work+GPU i7 8700 12threads
School i7 4770 8threads
Default+GPU Ryzen 7 3700X 16threads
Ryzen 7 3800X 16 threads
Ryzen 9 3900X 24threads
Home i7 3540M 4threads50%
[Nov 14, 2019 1:29:22 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Jean-David Beyer
Senior Cruncher
USA
Joined: Oct 2, 2007
Post Count: 339
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
I have been running N216 (high resolution) climateprediction models and here is what it uploaded. Surely they would not be uploading uncompressed .zip files, would they?

Tue 03 Dec 2019 09:35:42 AM EST | climateprediction.net | Started upload of hadam4h_a1vb_201311_4_842_011906879_1_r1054874838_4.zip 
Tue 03 Dec 2019 09:47:13 AM EST | climateprediction.net | Finished upload of hadam4h_a1vb_201311_4_842_011906879_1_r1054874838_4.zip
Tue 03 Dec 2019 10:52:05 AM EST | climateprediction.net | Started upload of hadam4h_a1vb_201311_4_842_011906879_1_r1054874838_out.zip
Tue 03 Dec 2019 10:52:09 AM EST | climateprediction.net | Finished upload of hadam4h_a1vb_201311_4_842_011906879_1_r1054874838_out.zip

----------------------------------------
[Dec 4, 2019 12:39:49 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7697
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
Surely they would not be uploading uncompressed .zip files, would they?

If it is a .zip file, it has already been compressed.
Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Dec 4, 2019 2:52:21 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 261
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available

If it is a .zip file, it has already been compressed.

Assuming the no-compression option wasn't invoked (e.g. -n suffixes or --suffixes suffixes using zip in Linux) smile - unlikely, of course, except for files already compressed such as .gif .arj etc where specifying this options saves time...
----------------------------------------
Run Time Stats https://grassmere-productions.no-ip.biz/
[Dec 4, 2019 3:16:16 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7697
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available

If it is a .zip file, it has already been compressed.

Assuming the no-compression option wasn't invoked (e.g. -n suffixes or --suffixes suffixes using zip in Linux) smile - unlikely, of course, except for files already compressed such as .gif .arj etc where specifying this options saves time...

It kind of begs the question of why you would create a zip file if it was not to compress it, especially if you were sending it. I suppose there are reasons which are valid, such as creating an archive, but I am just old fashioned enough to equate a zip file with compression even though that is not the only reason for the .zip format. (My apologies to Phil Katz.)
Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Dec 4, 2019 4:01:34 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 49   Pages: 5   [ Previous Page | 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread