| Index | Recent Threads | Unanswered Threads | Who's Active | Guidelines | Search |
 |
World Community Grid Forums
Category: Official Messages
Forum: News
Thread: 2022-08-26 Update (increased WU output & WCG backend changes) |
| No member browsing this thread |
|
Thread Status: Active Total posts in this thread: 98
|
|
| Author |
|
|
mdxi
Advanced Cruncher Joined: Dec 6, 2017 Post Count: 109 Status: Offline Project Badges: 
|
I've been getting WUs all through the week, but the rate of those WUs has decreased dramatically.
----------------------------------------A week ago I was crunching close to 200 WUs / day / machine. In the past 24 hours my machines have done an average of 8 WUs.  |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1951 Status: Offline Project Badges: 
|
Am I the only one getting a continuous stream of WUs? I have opted out of ARP so I can't comment on that. After a brief hiccup the downloading seems to be back to normal again and other than the Results problem everything appears fine. Nothing is fine, as the results page itself is the only thing beside the forum that is actually "working" on the whole web site.And there has been no "continuous stream of WUs" since at least Friday, all that has come in since are re-sends... Work done is definitively being registered and this can be seen in WCG Statistics (two along from Tasks). My User Total has gone up from 15,430,000 on 07 July to 15,545,000 today but I've no idea what these numbers represent - can anyone tell me? . I'm hoping that these figures can be translated into Points at some time when the whole thing is running normally again. Well, that is the problem right there. What you see in the BOINC client are the "points" that WCG is making available to the external stats sites. However, there is no accumulation of any of that data (not only points, but also WU count, and accumulated runtime per project) on the WCG web site. Hasn't been ever since they started to send out "test" WUs back in June. And bar any definitive answer from WCG/Krembil, all the WCG related data is at this point to be assumed lost. There has only been a somewhat generalized statement that all the science results are in fact being send to the researches of the respective projects, and we just have to take their word for it...Ralf  |
||
|
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2166 Status: Offline Project Badges: 
|
What you see in the BOINC client are the "points" that WCG is making available to the external stats sites. They are called credit(s), not points. WCG Points = credit multiplied by 7.However, there is no accumulation of any of that data […] on the WCG web site. Your statement is actually false; the "points" - as you call it - can be found - as credit - and are being made available in a certain directory meant for - you mentioned it yourself - external sites and that directory is part of the WCG website as well.And bar any definitive answer from WCG/Krembil, all the WCG related data is at this point to be assumed lost. What do you make of Cyclops' post that was posted two hours before yours, Ralf?I quote from that post: Fixing the contribution page to reflect all of the work done within the testing phase is on the tech team's radar, so we will make an announcement when it is fixed. [Edit 2 times, last edit by adriverhoef at Sep 6, 2022 8:13:49 PM] |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1951 Status: Offline Project Badges:
|
However, there is no accumulation of any of that data […] on the WCG web site. Your statement is actually false; the "points" - as you call it - can be found - as credit - and are being made available in a certain directory meant for - you said it yourself - external sites and that directory is part of the WCG website as well.There is not, anywhere, an up to date accumulation of any statistics, be it points/credits or WU count and runtime, separated by project within WCG. The "My project contribution" section of the Overview page has not been update since February 22, 2022! That's a fact! And bar any definitive answer from WCG/Krembil, all the WCG related data is at this point to be assumed lost. What do you make of Cyclops' post that was posted two hours before yours, Ralf?I quote from that post: Fixing the contribution page to reflect all of the work done within the testing phase is on the tech team's radar, so we will make an announcement when it is fixed. Please stop pretending that everything is just hunky-dory, it isn't! Ralf |
||
|
|
Jean-David Beyer
Senior Cruncher USA Joined: Oct 2, 2007 Post Count: 337 Status: Offline Project Badges:
|
Am I the only one getting a continuous stream of WUs? I have opted out of ARP so I can't comment on that. After a brief hiccup the downloading seems to be back to normal again and other than the Results problem everything appears fine. You may not be the only one, but I am not one. I got only one (MCM) work unit today although I am quite willing to take at least three work units for each of the five historically available ones. I did rather well last week, but not this week.  |
||
|
|
alanb1951
Veteran Cruncher Joined: Jan 20, 2006 Post Count: 964 Status: Offline Project Badges:
|
The longer I see this lack of progress on user profile statistics going on, the more I wonder if it isn't all down to the promised attempt to "conserve the streak", a statistic that was presumably put together by the same process(es) that did the now-unchanging local user profile data...
----------------------------------------Stating the obvious, at WCG there are two sorts of statistics gathering, one for the stuff that goes to places like BOINCStats and one for the WCG Profile pages and "My Contribution" totals. Perhaps slightly less obvious is that those two gatherings are not done by the same server processes[1]... So it is possible that "conserving the streak" without putting in a lot of coding effort and possible temporary database restructuring might have been done by simply turning off the profile statistics gathering on the assumption that all would've been sorted out by the time serious use of the site resumed. (Unfortunately, we all know that [whatever the reasons], the move turned out to be a lot more complex than expected, but it's futile re-hashing arguments about that...) If it really is that simple, I'd say just turn the stats runs back on and too bad about the [lost] streak... But perhaps it's not that simple after all - perhaps the extra stats runs are still being done but the output is going somewhere off to one side, waiting to be merged back into the "live" data; it is left as an exercise for database-savvy readers to envisage what could go wrong with that merge :-) ... [1] Folks may remember the twice-daily Statistics runs (at about 00:06 UTC and 12:06 UTC) on the WCG site in IBM days -- as far as I am aware, that's when the profile statistics were accumulated, as the "My Contribution" valies always changed shortly thereafter... Cheers - Al. P.S. Adri, Ralf; please stop shouting past one another :-) [Edit 1 times, last edit by alanb1951 at Sep 6, 2022 9:26:50 PM] |
||
|
|
adriverhoef
Master Cruncher The Netherlands Joined: Apr 3, 2009 Post Count: 2166 Status: Offline Project Badges:
|
However, there is no accumulation of any of that data […] on the WCG web site. Your statement is actually false; the "points" - as you call it - can be found - as credit - and are being made available in a certain directory meant for - you said it yourself - external sites and that directory is part of the WCG website as well.… of all WCG projects combined, for one particular day. No, they are the total WCG credit compiled at a certain moment of the day, not for one particular day.After the external stats sites pick up that data, it's gone/stale. The file is being updated, only a few times a day, if that is what you meant, then you've got that right.There is not, anywhere, an up to date accumulation of any statistics, be it points/credits or WU count and runtime, separated by project within WCG. The "My project contribution" section of the Overview page has not been update since February 22, 2022! That's a fact! There is a total WCG credit in a file in a certain directory, that is what I have been stating all the time, not anything else. Let me be more precise …If you download that file, you can see your total WCG credit at the time the file was compiled. I have downloaded the file, of course, it has a certain timestamp, and I will show you my statistics from that file and they are my total WCG credits at the time the file was compiled. I'm just showing you what I'm telling you. I am claiming nothing else. $ ls -l user.gz There you have it, 147531486.667685 being my total credit, easy to spot. Nothing more, like I claimed all the time, nothing else. Please stop pretending that everything is just hunky-dory, it isn't! It seems you are addressing me, but I don't understand why you feel like I'm pretending everything is hunky-dory. Haven't you seen any posts from me, identifying issues?One of the first issues I identified were the missing images and one of the last was something else. Is everything hunky-dory in your eyes when I have been identifying issues that haven't been resolved? Not by my standards. Do I pretend or have I been pretending everything is hunky-dory? Not that I know. EDIT: Hi Al, sorry, didn't see your post until after I posted this one.  EDIT 2: Fixed a quotation error and fixed my statement about multiplication by 7. [Edit 4 times, last edit by adriverhoef at Sep 7, 2022 9:53:47 AM] |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1951 Status: Offline Project Badges:
|
The longer I see this lack of progress on user profile statistics going on, the more I wonder if it isn't all down to the promised attempt to "conserve the streak", a statistic that was presumably put together by the same process(es) that did the now-unchanging local user profile data... Absolutely NOT! That "streak" thing that came upon us with that stupid web site redesign just before the move, and is probably the worst (as in most useless) statistic of all. Anyone who is crunching for more than just one day will have a continuous "streak". Absolutely pointless. The same goes pretty much for the "heat map" right next to it.Stating the obvious, at WCG there are two sorts of statistics gathering, one for the stuff that goes to places like BOINCStats and one for the WCG Profile pages and "My Contribution" totals. There "used to be two sort of statistics", as the WCG internal stats don't work (anymore) Perhaps slightly less obvious is that those two gatherings are not done by the same server processes[1]... So it is possible that "conserving the streak" without putting in a lot of coding effort and possible temporary database restructuring might have been done by simply turning off the profile statistics gathering on the assumption that all would've been sorted out by the time serious use of the site resumed. And the streak thingy doesn't work. At some point (when the internal WCG project stats already weren't working, after the first test WUs were returned back in June), they (someone at Krembil/WCG) tried to put a "fake" prolonged streak on the web site, which just kept counting regardless if test WUs were returned or not. And then that was removed/disabled, and the "current" streak now shows again 0 (zero) for weeks, despite more and more WUs being returned. (Unfortunately, we all know that [whatever the reasons], the move turned out to be a lot more complex than expected, but it's futile re-hashing arguments about that...) As I mentioned, those internal stats relied on the WUs being validated and then processed to, for example, produce the matching badges.If it really is that simple, I'd say just turn the stats runs back on and too bad about the [lost] streak... But perhaps it's not that simple after all - perhaps the extra stats runs are still being done but the output is going somewhere off to one side, waiting to be merged back into the "live" data; it is left as an exercise for database-savvy readers to envisage what could go wrong with that merge :-) ... My wild guess, based on half a sentence that Cyclops made a while go, is that the server that is actually processing those stats is not among those that have been turned back on yet. Hence no stats. Now the downsite on that is likely that with the validated WUs being purged after a couple of days out of the "Results" database, that server won't be able to retroactively process all those (now purged) WUs in order to update the stats. Thus all that work (from the POV of the people crunching, still assuming that we can take WCG/Krembil's word for it that the scientific results are being forwarded to the researchers) will be lost until they re-establish the internal stats process. In my case, so far (I have only started to take note of the daily results since 8/23), this amounts for +4k of results returned, with +13 months of combined runtime (CPU time), all being lost. [1] Folks may remember the twice-daily Statistics runs (at about 00:06 UTC and 12:06 UTC) on the WCG site in IBM days -- as far as I am aware, that's when the profile statistics were accumulated, as the "My Contribution" valies always changed shortly thereafter... Correct. That is the part that is no longer working.Ralf ---------------------------------------- |
||
|
|
cappucino
Cruncher Joined: May 5, 2007 Post Count: 17 Status: Offline Project Badges:
|
On my PC I've not seen nor crunched a single wu since last week, as nothing was coming through. I decided it best to add in TN-Grid due to the unreliability of WCG wu's being sent. I'm not really into tweaking BOINC to the point of keeping a core open and wait for WCG to eventually get something to me successfully, so until can be back at 100% I'm sticking with TN-Grid as they seem like they can keep more cores busy.
----------------------------------------By the same token my Android tablet has been kept busy sending over WCG wu's, so I'm still doing work for them exclusively on that device and that one stays up 24/7/365, unlike my PC that gets a nap when I sleep. At this point I feel I participated in a two week beta period where I *thought* that Krembil had all their issues sorted out and they just needed to go to production and be live. Obviously that's not the case. nivrip, you're getting a steady stream of all project wu's with no issues? You must have tapped into those "Yorkshire Domes" in your neck of the woods?  I just wrapped up a big project that involved a few people from the UK, the US and Spain of a (bootleg) recording done at the Barbican back in April of 2022. A fun little test to prove that the merits of live recording on old technology can actually sound just as good, if not better than on some of the most recent digital technology. Yes, it was found out recording onto old metal cassette tapes can be jaw dropping and probably hasn't been done in over 20 years or more. [Edit 1 times, last edit by cappucino at Sep 7, 2022 1:36:28 AM] |
||
|
|
TPCBF
Master Cruncher USA Joined: Jan 2, 2011 Post Count: 1951 Status: Offline Project Badges:
|
]No, they are the (total) credit, not the points. Credit = WCG Points multiplied by 7.  I think you got something mixed up here due to your semantics games, but WCG points are 7x the amount of BOINC points (point, credits, same difference, just semantics).… of all WCG projects combined, for one particular day. No, they are the total WCG credit compiled at a certain moment of the day, not for one particular day.There is not, anywhere, an up to date accumulation of any statistics, be it points/credits or WU count and runtime, separated by project within WCG. The "My project contribution" section of the Overview page has not been update since February 22, 2022! That's a fact! There is a total WCG credit in a file in a certain directory, that is what I have been stating all the time, not anything else. Let me be more precise …[/quoteRalf |
||
|
|
|