Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
![]() |
World Community Grid Forums
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
No member browsing this thread |
Thread Status: Active Total posts in this thread: 98
|
![]() |
Author |
|
alanb1951
Veteran Cruncher Joined: Jan 20, 2006 Post Count: 972 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Ralf,
The streak has to be calculated somehow as in a standard BOINC database (which I doubt IBM tampered with...) the "work done today" numbers are only held in the host_app_version table and the host table, not the user table, so the obvious way to do it was to tack it on to the existing WCG profile stats run. As for reasons for not running said stats, it may or may not be connected... That's the only reason for my "whimsy" about the streak being the issue here - I obviously didn't apply enough smileys to it! And I wonder if either Adri or you read this post from a month or so ago about internal and external statistics and the likely outcomes when they [try to] turn the internal stats stuff back on again; it was in response to a pair of your postings, but neither of you commented further (which might have been a blessing!) :-) Cheers - Al. |
||
|
wildhagen
Veteran Cruncher The Netherlands Joined: Jun 5, 2009 Post Count: 846 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Personally, I don't care about those streaks, as they are useless by design. Two days crunching in a row, and you're on a streak. Wow... i'm impressed. Or not.
I'd rather see the much more important 'normal' stats online. |
||
|
nivrip
Senior Cruncher North Yorkshire Joined: Sep 13, 2007 Post Count: 264 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
[quote nivrip, you're getting a steady stream of all project wu's with no issues? You must have tapped into those "Yorkshire Domes" in your neck of the woods? ![]() . Spoke too soon !!! Only getting MCMs now and nothing else. But still a continuous stream. ![]() ----------------------------------------
ЮРКШИР КРУНЧЕР
|
||
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2167 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Al:
----------------------------------------And I wonder if (…) Adri (…) read this post from a month or so ago about internal and external statistics and the likely outcomes when they [try to] turn the internal stats stuff back on again; it was in response to a pair of your postings, but neither of you commented further (which might have been a blessing!) :-) Cheers - Al. I did, think at least must have missed your 10th August edit, and didn't feel the need to comment (as there was a lot of speculation, if you must know ![]() [Edit 2 times, last edit by adriverhoef at Sep 7, 2022 10:18:24 AM] |
||
|
Ingleside
Veteran Cruncher Norway Joined: Nov 19, 2005 Post Count: 974 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Now the downsite on that is likely that with the validated WUs being purged after a couple of days out of the "Results" database, that server won't be able to retroactively process all those (now purged) WUs in order to update the stats. A quick search did find the documentation of BOINC's Database Purging Utility (db_purge), and this says: BOINC provides a utility db_purge that writes result and WU records to XML-format archive files, then deletes them from the database. Meaning, for the WCG-specific stats it should be possible to parse the db_purge archive files and this way nothing is really lost. While it would definitely be nice to see something besides zeros in my current production stats, WCG being back to 21+ hours deferral due to lack of work is at least to me a much more pressing problem than knowing if I've hit another badge. ![]() "I make so many mistakes. But then just think of all the mistakes I don't make, although I might." |
||
|
thunder7
Senior Cruncher Netherlands Joined: Mar 6, 2013 Post Count: 232 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Dear volunteers, We have taken additional measures to increase the quantity of WUs we can send out, and we have been able to increase the quantity of WUs in flight at any given time. Volunteers should see this reflected on their devices now, and perhaps even over this past week. We are also relieved to share that the hosting data centre has assigned additional personnel on site to resolve our networking issues, meaning a fix is imminent. We will share with you any further updates we receive from the data centre. The network fix will allow us to bring our remaining servers online, stabilizing and further increasing the WU supply. Thus, until we are able to deploy all dedicated servers, we must continuously adjust and monitor tasks scheduled in Aurora/Mesos to keep the tasks balanced and the workunits flowing, and so far this process is unduly intensive and sporadic. For example, a recurring job may saturate the scheduler by creating a large number of downstream jobs. This flood of new jobs might then throttle the processing rate of other waiting jobs and thereby interrupt the supply of work. To fix the problem, we would need to temporarily deschedule the parent job, decrease its frequency, or decrease the priority of its children in such a way that does not starve other stages of the pipeline. Last week, we mentioned that we have begun to investigate concerns over statistics, credit, streaks, and database dumps raised by volunteers. We will have an update on some of these issues next week. We also plan to release a more structured breakdown from the tech team similar to a CHANGELOG starting next week or the week after so that we can increase the frequency and clarity of updates. Thanks for your patience and have a great weekend! -WCG Tech Team So, some points for the (overdue) update: 1) The quantity of WU's is still spotty. What's happening and when will this be fixed? 2) Did the data centre not send any updates over the last 10 days, which you would have shared? 3) Did that network fix happen? If so, what are the results? 4) You promised an update on concerns over statistics, credits etc. Did anything happen? If so, what? 5) So how are things with that CHANGELOG coming along? We are now in 'the week after that'... 6) Thanks for the wish for a great weekend. Indeed, I had 2 successively and still no update? All in all, would you concur that communication is a bit lacking, both in frequency and in promised updates which don't happen? [Edit 1 times, last edit by thunder7 at Sep 7, 2022 3:14:33 PM] |
||
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1951 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
And I wonder if either Adri or you read this post from a month or so ago about internal and external statistics and the likely outcomes when they [try to] turn the internal stats stuff back on again; it was in response to a pair of your postings, but neither of you commented further (which might have been a blessing!) :-). Well, that's becausea) it is not a post from a technical person at WCG/Krembil given some insights as to what is happening (or not). b) is actually not explaining even from an outside view what is happening, or not. Something either has been thoroughly been borked during the transition, or the actual reason might be as simple as not having the server powered back up that runs the twice daily internal WCG stats.. And that run is independent, apparently not related to the external (BOINC) stats file creation, which runs (ran) at different times. And in the past, I don't recall ever that there were problems with the internal stats generation, beside if there was a (un)planned outage due to database upgrade/maintenance and knreed and/or uplinger had to kick the process that created those external stats file back into gear. Again, never was there an issue with the internal stats. And as you can see, I am with the WCG project not just since yesterday and I do have returned quite an amount of work for the various project during this time.That combined with the fact that I am working in IT, doing sysadmin/database/programming work for a few decades by now, I think I can make observations and deductions as to what may or may not be at least the source of a problem. The real big issue here is that Krembil is absolutely lacking any transparency about what is happening.. I have been sticking with WCG for all those years because in the past, this was the most well run DC project out there. Very rarely having real issues, and if something happened, which according to Mr.Murphy is always a possibility, things were sorted rather quickly, usually in a matter of a couple of days, if not hours. Rosetta's Baker Lab at WU has gone though problems for example as well, but never have I seen such a mess (to put it simply) as currently with WCG at Krembil.There are single dude projects run from a garage that are better organized and reliable. The download issues "seemed" to be solved a few days ago, now it's all back as bad as it ever was. No discernible progress.Likewise with the stats issue. What some people forget is the overall time frame we are dealing with here. The move was announced 1 year ago! (And no, I don't feel like celebrating). It was 5 months before the system was shutdown at IBM and the physical transition to Krembil was supposed to start. It was another 4 months until the first "Test WUs" were being send out. And now they are "testing" for another. 3 months, with nothing really to show for. And Krembil is not even able to provide regular updates, with some real progress (or problems) info, just that eventually someone will look into an issue, they do have it on their radar... ![]() And no, I am not the only one complaining, maybe just the most vocal. Not to mention all those newcomers, people that have a real interest in the science behind the WCG projects, that are being put off and leave before you even saw anything from them... Ralf ![]() ![]() |
||
|
alanb1951
Veteran Cruncher Joined: Jan 20, 2006 Post Count: 972 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Well, that's me put in my place :-)
I'll go back to being a retired programmer, database admin (et cetera) who wishes he could do more to help but [as an outsider with no eyes on WCG's extra code] cannot do so. Cheers - Al. |
||
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2167 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Ralf:
I don't recall ever that there were problems with the internal stats generation, Oh no, not again! ![]() ![]() beside if there was a (un)planned outage due to database upgrade/maintenance Right.Again, never was there an issue with the internal stats. I'm afraid there was. ![]() Three years ago, something unusual happened with the internal WCG stats. Adri |
||
|
Sgt.Joe
Ace Cruncher USA Joined: Jul 4, 2006 Post Count: 7677 Status: Offline Project Badges: ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
Well, that's me put in my place :-) I'll go back to being a retired programmer, database admin (et cetera) who wishes he could do more to help but [as an outsider with no eyes on WCG's extra code] cannot do so. Cheers - Al. You offer insight - keep it up. Cheers
Sgt. Joe
*Minnesota Crunchers* |
||
|
|
![]() |