| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Active Research
Forum: Africa Rainfall Project
Thread: BIG file size! Hope you have a lot of bandwidth available |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 49
|
|
| Author |
|
|
deltavee
Ace Cruncher Texas Hill Country Joined: Nov 17, 2004 Post Count: 4891 Status: Offline Project Badges: 
|
Not overly computer savvy so bear with me... Ok, they're big files. Other than the 1-2 times per day that they spend a few minutes down/uploading the files, is the project any different crunching wise than any other project? I'm not really seeing the issue other than the fact that you're dealing with big files (ok, if I was crunching on a smart phone & had a limited data plan...yeah, that could be an issue). On the computer I'm on now there are six cores and plenty of ram available & I usually have the Grid grinding away in the background with no issues. Would making say three of those cores available for the Africa project impact the computer's performance? Other than a lower 'Results Returned' score, 24 hours of crunching is 24 hours of crunching... or am I missing something here? Greg L, I think you are more savvy than you think. On one cruncher I am running all out on all threads without any problem. If you can currently crunch WCG and have enough ram you should be okay. [Edit 1 times, last edit by deltavee at Oct 31, 2019 5:12:52 PM] |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
The bandwidth per CPU hour of crunching is not that exceptional. 60MB upload is 2.5MB per hour computing. Just look at what for instance MIP1 tasks consume by comparison.
----------------------------------------[Edit 1 times, last edit by Former Member at Oct 31, 2019 7:22:03 PM] |
||
|
|
hchc
Veteran Cruncher USA Joined: Aug 15, 2006 Post Count: 811 Status: Offline Project Badges: 
|
JcJet said:
----------------------------------------No big deal. It's notmal that data size rise with time as new projects apearing. And it's acutually means GOOD things. For science, not for someone's cellular internet. Because in science, the method or algorithm itself it's only half of the possible solution, the other half is dataset you use to 'mine' for new data. And if datasets are too small or have flaws like biases, it will affect result a lot. There are fields which stronlgy limited by the lack of bigger, broader datasets (even though the current ones are several TBs in size). So bigger datasets = good. I know this is your first post to this forum (welcome!), but it's comparing apples to oranges to say that a project with a bigger dataset is inherently better than another project with a smaller dataset. Completely apples to oranges. One could speculate that a project that deals with images (like weather/climate to medical imaging) will have larger storage requirements than a project that looks for the lowest energy affinities of a chemical compound to a protein target, for example. Images and GIS data are simply larger by their nature.
|
||
|
|
JcJet
Cruncher Joined: Oct 5, 2010 Post Count: 3 Status: Offline Project Badges: 
|
JcJet said: No big deal. It's notmal that data size rise with time as new projects apearing. And it's acutually means GOOD things. For science, not for someone's cellular internet. Because in science, the method or algorithm itself it's only half of the possible solution, the other half is dataset you use to 'mine' for new data. And if datasets are too small or have flaws like biases, it will affect result a lot. There are fields which stronlgy limited by the lack of bigger, broader datasets (even though the current ones are several TBs in size). So bigger datasets = good. I know this is your first post to this forum (welcome!), but it's comparing apples to oranges to say that a project with a bigger dataset is inherently better than another project with a smaller dataset. Completely apples to oranges. One could speculate that a project that deals with images (like weather/climate to medical imaging) will have larger storage requirements than a project that looks for the lowest energy affinities of a chemical compound to a protein target, for example. Images and GIS data are simply larger by their nature. Maybe I wasn't clear enough, sorry. I'm not comparing computer vision datasets to others. I'm saying that for the same task and method, the bigger dataset is _usually_ better. For example, more detailed or broader climate data. |
||
|
|
catchercradle
Advanced Cruncher England Joined: Jan 16, 2009 Post Count: 133 Status: Offline Project Badges: 
|
Yes climate apps have a lot of data involved as those of us who regularly crunch for Climate prediction.net know. Still haven't managed to get any of these but at least they are not like one of the recent beta apps for CPDN where uploads were over 1GB!
|
||
|
|
DCS1955
Veteran Cruncher USA Joined: May 24, 2016 Post Count: 668 Status: Offline Project Badges:
|
Yes climate apps have a lot of data involved as those of us who regularly crunch for Climate prediction.net know. Still haven't managed to get any of these but at least they are not like one of the recent beta apps for CPDN where uploads were over 1GB! 1GB!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Wow they need to be a little more granular.   |
||
|
|
hchc
Veteran Cruncher USA Joined: Aug 15, 2006 Post Count: 811 Status: Offline Project Badges:
|
Yes climate apps have a lot of data involved as those of us who regularly crunch for Climate prediction.net know. Still haven't managed to get any of these but at least they are not like one of the recent beta apps for CPDN where uploads were over 1GB! 1GB!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Wow they need to be a little more granular. Wonder if they even compress output before uploads? 1 GB compressed is massive. 1 GB uncompressed can probably be squeezed a bit smaller.
|
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Wonder if they even compress output before uploads? 1 GB compressed is massive. 1 GB uncompressed can probably be squeezed a bit smaller. The penultimate line of the result log is: INFO: Simulation complete compressing output.so I think you have your answer. |
||
|
|
Jack007
Master Cruncher CANADA Joined: Feb 25, 2005 Post Count: 1604 Status: Offline Project Badges: 
|
Interesting on the ram, I have 16 gigs, and with windows, 10 MIP, and 4 Africa running, only using 6 gigs. Just in case I ever get a LOT of Africa running, I just bought 16 more gigs. I think it's all DDR4 3000 RAM.
---------------------------------------- |
||
|
|
hchc
Veteran Cruncher USA Joined: Aug 15, 2006 Post Count: 811 Status: Offline Project Badges:
|
Apis said:
----------------------------------------Wonder if they even compress output before uploads? 1 GB compressed is massive. 1 GB uncompressed can probably be squeezed a bit smaller. The penultimate line of the result log is: INFO: Simulation complete compressing output.so I think you have your answer. I was talking about the Climateprediction.net 1 GB uploads. I only do WCG so I wouldn't know.
|
||
|
|
|