| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Active Research
Forum: Africa Rainfall Project
Thread: Welcome to the Africa Rainfall Project |
| No member browsing this thread |
|
Thread Status: Active Thread Type: Sticky Thread Total posts in this thread: 68
|
|
| Author |
|
|
hchc
Veteran Cruncher USA Joined: Aug 15, 2006 Post Count: 802 Status: Offline Project Badges: 
|
uplinger said:
----------------------------------------I appreciate the friendly sarcasm. The reality is, we would love to run this at full speed as well. With this project being a beast, it takes a lot to make sure things are processing properly. We are making sure our hosting can handle it with the current setup or if major updates are required to support it. We are also working with the researchers to make sure the data get put in the proper location for result analysis. I had an estimate from early on in this project that we would exceed 1 PByte of data for the scope it is now. Storing this information for say 10 years is not cheap as you need replication to prevent data loss. Also this information needs to be easily and quickly accessed for analysis. Until some of this is fully resolved, we are processing at the current speed with incremental updates as World Community Grid confirms we don't need to upgrade anything. Thanks, -Uplinger Wow, 1 Petabyte estimate. I was wondering about some of the nerdy IT details. I've seen some research groups use Amazon's AWS for long-term storage and data analysis -- can't remember if they were former WCG projects. I assume IBM is chipping in with IBM Cloud for the time being, but I don't know the long-term agreement. That's one heck of a SQL database if y'all are using some flavor of SQL. I'd freak out if it wasn't backed out and replicated through the wazoo and some disaster hits.
|
||
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7670 Status: Offline Project Badges: 
|
Wow, 1 Petabyte estimate. I was wondering about some of the nerdy IT details. No matter which way you cut it, that is a substantial amount of storage. I presume that is the estimated storage requirement for the potential life of the project. Just musing again on a potential graphic which would indicate the progress of the project. Perhaps a grid overlay of the area being tested with each subdivision of the grid representing the work unit portion being tested. With 182 iterations of the area being run each grid square (?) could be color coded after each 25 iterations for that area, so you have 8 different colors for an area progressing until that area is finished. With that kind of visual, with a snapshot of the progress every 24 hours would make a nice time lapse record of how the project proceeded. I would not expect WCG to attempt this, but maybe the scientists could get some hotshot grad student to program this type of visual. If a person actually had access to the grid, there may be some off the shelf statistical software which may provide this type of functionality. This may be a potential public relations enhancement which could be used by the scientists to use when promoting their project. Cheers
Sgt. Joe
----------------------------------------*Minnesota Crunchers* [Edit 1 times, last edit by Sgt.Joe at Nov 9, 2019 3:46:35 PM] |
||
|
|
apollinaris
Cruncher Joined: Sep 1, 2015 Post Count: 4 Status: Offline Project Badges:
|
jo
|
||
|
|
Jack007
Master Cruncher CANADA Joined: Feb 25, 2005 Post Count: 1604 Status: Offline Project Badges: 
|
Hmm about us storing for them.
----------------------------------------I know how often I lose/replace hard drives, and I remember some project that was gonna archive all print media if I remember right. Just don't think it's practical.  |
||
|
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7670 Status: Offline Project Badges:
|
Hmm about us storing for them. I know how often I lose/replace hard drives, and I remember some project that was gonna archive all print media if I remember right. Just don't think it's practical. Potentially, that much storage gets a little expensive. According to this outfit it would cost about $60,000 to set up a petabyte of storage. Not an insignificant sum. Of course you would want to build in some redundancy, so you don't lose any data, so I would probably double that cost. Then of course there is the cost of software, cooling, electricity, etc. I trust the funding is in place for something like this. Maybe IBM has a cheaper solution. Cheers
Sgt. Joe
----------------------------------------*Minnesota Crunchers* [Edit 2 times, last edit by Sgt.Joe at Nov 10, 2019 2:39:22 AM] |
||
|
|
Seoulpowergrid
Veteran Cruncher Joined: Apr 12, 2013 Post Count: 817 Status: Offline Project Badges: 
|
Just musing again on a potential graphic which would indicate the progress of the project. Perhaps a grid overlay of the area being tested with each subdivision of the grid representing the work unit portion being tested. With 182 iterations of the area being run each grid square (?) could be color coded after each 25 iterations for that area, so you have 8 different colors for an area progressing until that area is finished. With that kind of visual, with a snapshot of the progress every 24 hours would make a nice time lapse record of how the project proceeded. I would not expect WCG to attempt this, but maybe the scientists could get some hotshot grad student to program this type of visual. If a person actually had access to the grid, there may be some off the shelf statistical software which may provide this type of functionality. This may be a potential public relations enhancement which could be used by the scientists to use when promoting their project. I love the way you think! That would be amazing to have and really encouraging.  |
||
|
|
uplinger
Former World Community Grid Tech Joined: May 23, 2005 Post Count: 3952 Status: Offline Project Badges: 
|
Very interesting have a new Project to colaborate, but in this case the tasks are very rare, and have a strange schedule in its distribution. I have several machines in random fetch asking for new tasks of African Climate and since de day of start don1t get anyone after the start day. Could you please Professor, ask for the WCG change the way of distribution putting a randomic distribution during teh 24 hours of the day, for this Project. This simple change could improve the number of entusiasta to help the Project today or in the future, once will provide mnore equity in the distribution criterias which today appear be based on specific times. Think about please. This idea have the pótential to help much more your Project.      Giba, I see what you are saying about the distribution. We are hopeful that the limitation on the speed of this project will cause this to be a non issue in the future. I'll look into randomizing the release, no promises as it'll need to be something simple to change since I hope we can run more at a time in the future. (Note: I do not have an eta on when that future is) Thanks, -Uplinger Giba, I have a solution to randomizing the distribution of results in testing at the moment on our test environment. I'll let you know when it is pushed to production. Thanks, -Uplinger |
||
|
|
Former Member
Cruncher Joined: May 22, 2018 Post Count: 0 Status: Offline |
Very interesting have a new Project to colaborate, but in this case the tasks are very rare, and have a strange schedule in its distribution. I have several machines in random fetch asking for new tasks of African Climate and since de day of start don1t get anyone after the start day. Could you please Professor, ask for the WCG change the way of distribution putting a randomic distribution during teh 24 hours of the day, for this Project. This simple change could improve the number of entusiasta to help the Project today or in the future, once will provide mnore equity in the distribution criterias which today appear be based on specific times. Think about please. This idea have the pótential to help much more your Project. Giba, I see what you are saying about the distribution. We are hopeful that the limitation on the speed of this project will cause this to be a non issue in the future. I'll look into randomizing the release, no promises as it'll need to be something simple to change since I hope we can run more at a time in the future. (Note: I do not have an eta on when that future is) Thanks, -Uplinger Giba, I have a solution to randomizing the distribution of results in testing at the moment on our test environment. I'll let you know when it is pushed to production. Thanks, -Uplinger While at it, kindly apply to HSTB too, and make it random to the second if that's possible. |
||
|
|
Aurum
Master Cruncher The Great Basin Joined: Dec 24, 2017 Post Count: 2386 Status: Offline Project Badges:
|
I had an estimate from early on in this project that we would exceed 1 PByte of data for the scope it is now. Storing this information for say 10 years is not cheap as you need replication to prevent data loss. Also this information needs to be easily and quickly accessed for analysis. You could use Storj Tardigrade distributed cloud storage and we can all help store it....we are processing at the current speed with incremental updates as World Community Grid confirms we don't need to upgrade anything. No idea what this phrase means. Do you mean WCG has identified no bugs as yet???Thanks, -Uplinger Thanks for your response.   ...KRI please cancel all shadow-banning |
||
|
|
Jack007
Master Cruncher CANADA Joined: Feb 25, 2005 Post Count: 1604 Status: Offline Project Badges:
|
hmmm well random times would probably get me more work units overall, it has been so rewarding when I do hit some on the 8s. Like a winning scratch ticket.
---------------------------------------- |
||
|
|
|