Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 
World Community Grid Forums
  Category: Active Research
  Forum: Africa Rainfall Project
  Thread: BIG file size! Hope you have a lot of bandwidth available

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 49
Posts: 49   Pages: 5   [ Previous Page | 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 45733 times and has 48 replies Next Thread
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 261
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
Getting a bit OT - but do quite a bit of Audio Visual work here and useful for combining and sending a selection of AV files belonging to a project. An example the files required to create DVD, files such as .mp3 .gif .mp4 .mp3 etc. Compression has minimal gain in any I have seen with files of this type. A few get larger with compression :-( when I investigated and takes longer...
Doubt though this is the case of result files uploaded ny BOINC...
----------------------------------------
Run Time Stats https://grassmere-productions.no-ip.biz/
----------------------------------------
[Edit 1 times, last edit by TonyEllis at Dec 4, 2019 4:53:10 AM]
[Dec 4, 2019 4:52:19 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7695
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
Getting a bit OT - but do quite a bit of Audio Visual work here and useful for combining and sending a selection of AV files belonging to a project. An example the files required to create DVD, files such as .mp3 .gif .mp4 .mp3 etc. Compression has minimal gain in any I have seen with files of this type. A few get larger with compression :-( when I investigated and takes longer...
Doubt though this is the case of result files uploaded ny BOINC...

I agree with you. Most of those files are already in compressed format, so compressing them with another format yields little or nothing in terms of further compression.
Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Dec 4, 2019 4:59:51 AM]   Link   Report threatening or abusive post: please login first  Go to top 
dango
Senior Cruncher
Joined: Jul 27, 2009
Post Count: 307
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
You in the USA are still living in year 1990 of IT :D
[Dec 4, 2019 8:13:58 AM]   Link   Report threatening or abusive post: please login first  Go to top 
hchc
Veteran Cruncher
USA
Joined: Aug 15, 2006
Post Count: 811
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
Different levels of compression in the zip option, and the higher the desired compression, the longer it takes, the larger dictionary/word size, and the more RAM it takes, which may not be available on older systems, so sometimes happy medium must be chosen. Or just the default compression option that a script runs.

Raw image files do tend to compress well, but I admit to knowing nothing about the WRF application and how it processes image and climate data.

I guess we could carefully monitor the size of an ARP1 slot when it reaches 99.000% and checkpoints, then pause it and take a snapshot of that slot, then suspend network so that the size of the output files can be added up and compared. Then calculate compression ratio or something.
----------------------------------------
  • i5-7500 (Kaby Lake, 4C/4T) @ 3.4 GHz
  • i5-4590 (Haswell, 4C/4T) @ 3.3 GHz
  • i5-3570 (Broadwell, 4C/4T) @ 3.4 GHz

----------------------------------------
[Edit 2 times, last edit by hchc at Dec 4, 2019 10:01:41 AM]
[Dec 4, 2019 9:41:13 AM]   Link   Report threatening or abusive post: please login first  Go to top 
hchc
Veteran Cruncher
USA
Joined: Aug 15, 2006
Post Count: 811
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
dango said:
You in the USA are still living in year 1990 of IT :D

USA is behind many parts of Europe and Asia for both speed and cost. In both Internet and mobile phone. Partly because of area and geography, partly because of monopolistic practices and corporate greed and no real competition to drive prices lower. And partly because each state can serve as its own legal island.

I'm jealous of cheap prices I hear about in e.g. UK or Germany or Switzerland, even Eastern Europe like Romania as well as South Korea, Japan, Taiwan, etc.

Corporations here bury copper coax cable or fiber cable and own that infrastructure and guard it with their souls [Edit, correction, ISP/telecom corporations have no souls, they guard infrastructure with their lawyers] and engage in bloody legal battles with their competition for monopoly of the land. So most of USA currently is stuck with high prices for Internet and mobile and slow speeds and lack of incentive for ISPs and telecom to improve service.

And plus rural towns and farms may have no coax copper or fiber and can cost $1 Million+ just to wire one house so they are stuck with ancient 2-wire or 4-wire POTS copper for aDSL maybe 1.5 Mbps - 3 Mbps downstream if they are close enough, else they must go with 56k dial-up! Or satellite "broadband" which is still ancient.
----------------------------------------
  • i5-7500 (Kaby Lake, 4C/4T) @ 3.4 GHz
  • i5-4590 (Haswell, 4C/4T) @ 3.3 GHz
  • i5-3570 (Broadwell, 4C/4T) @ 3.4 GHz

----------------------------------------
[Edit 6 times, last edit by hchc at Dec 4, 2019 9:59:14 AM]
[Dec 4, 2019 9:45:55 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Jean-David Beyer
Senior Cruncher
USA
Joined: Oct 2, 2007
Post Count: 338
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
You in the USA are still living in year 1990 of IT :D


Not me: I estimate I am living in about 2009 of IT. ;-)
----------------------------------------
[Dec 4, 2019 2:18:57 PM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2171
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
If it is a .zip file, it has already been compressed.
Speaking of which, in ~boinc/slots/* nothing is what it seems …
[www.worldcommunitygrid.org]# ls -l ../../slots/13/wrfout_d0[123].zip
-rw-r--r--. 1 boinc boinc 97 Dec  4 04:15 ../../slots/13/wrfout_d01.zip
-rw-r--r--. 1 boinc boinc 97 Dec  4 04:15 ../../slots/13/wrfout_d02.zip
-rw-r--r--. 1 boinc boinc 97 Dec  4 04:15 ../../slots/13/wrfout_d03.zip
[www.worldcommunitygrid.org]# cat ../../slots/13/wrfout_d0[123].zip
<soft_link>../../projects/www.worldcommunitygrid.org/ARP1_0033056_000_0_r499808026_0</soft_link>
<soft_link>../../projects/www.worldcommunitygrid.org/ARP1_0033056_000_0_r499808026_1</soft_link>
<soft_link>../../projects/www.worldcommunitygrid.org/ARP1_0033056_000_0_r499808026_2</soft_link>
devilish
[Dec 4, 2019 2:37:18 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Sgt.Joe
Ace Cruncher
USA
Joined: Jul 4, 2006
Post Count: 7695
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
You in the USA are still living in year 1990 of IT :D

so they are stuck with ancient 2-wire or 4-wire POTS copper for aDSL maybe 1.5 Mbps - 3 Mbps downstream if they are close enough

I am one of those who are on the "ancient" POTS copper" lines for DSL. It is kind of ironic that this is the best service I can get because I have 2 separate fiber lines running not 20 feet from my house and also another fiber line running on the other side of the road from me. I have tried to get hooked up to any one of them and run into a lot of regulatory gobbledygook about how it is just not possible at this time. Not to mention what the probable cost would be. However, what I have now is way better than the 14.4 baud modem with which I started. The DSL is very dependable and it does handle all the data that gets thrown at it. So far there have been no limits on the data.
Cheers
----------------------------------------
Sgt. Joe
*Minnesota Crunchers*
[Dec 4, 2019 3:03:10 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
If it is a .zip file, it has already been compressed.
Speaking of which, in ~boinc/slots/* nothing is what it seems …
[www.worldcommunitygrid.org]# ls -l ../../slots/13/wrfout_d0[123].zip
-rw-r--r--. 1 boinc boinc 97 Dec  4 04:15 ../../slots/13/wrfout_d01.zip
-rw-r--r--. 1 boinc boinc 97 Dec  4 04:15 ../../slots/13/wrfout_d02.zip
-rw-r--r--. 1 boinc boinc 97 Dec  4 04:15 ../../slots/13/wrfout_d03.zip
[www.worldcommunitygrid.org]# cat ../../slots/13/wrfout_d0[123].zip
<soft_link>../../projects/www.worldcommunitygrid.org/ARP1_0033056_000_0_r499808026_0</soft_link>
<soft_link>../../projects/www.worldcommunitygrid.org/ARP1_0033056_000_0_r499808026_1</soft_link>
<soft_link>../../projects/www.worldcommunitygrid.org/ARP1_0033056_000_0_r499808026_2</soft_link>
devilish

LoL, a method WCG has been deploying on multiple projects if not all by now to minimize having N copies of the identical on the drive.
[Dec 4, 2019 6:26:41 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Crystal Pellet
Veteran Cruncher
Joined: May 21, 2008
Post Count: 1323
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: BIG file size! Hope you have a lot of bandwidth available
Example of 2 uploaded compressed result files:

ARP1_0033835_000_0_r1280440746_0
13,298,039 bytes compressed 31%
41,894,918 bytes decompressed

ARP1_0033835_000_0_r1280440746_1
12,285,332 bytes compressed 29%
41,894,918 bytes decompressed
[Dec 4, 2019 6:53:11 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 49   Pages: 5   [ Previous Page | 1 2 3 4 5 | Next Page ]
[ Jump to Last Post ]
Post new Thread