Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 15
Posts: 15   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 2529 times and has 14 replies Next Thread
cjslman
Master Cruncher
Mexico
Joined: Nov 23, 2004
Post Count: 2082
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Indicate which validated WU have been credited

Suggestion: implement an easy visual way of indicating which validated WUs have been credited to the cruncher. This could be done easily either by color coding the WUs or by adding a checkmark someplace or by adding a timestamp to the WU status page/popup. This would help greatly in understanding results/stats that the crunchers see ("Why did I get credited with so few hours?"... "WOW !!! 3 days of results credited? Where did that come from ?")

Thanks,

CJSL
----------------------------------------
I follow the Gimli philosophy: "Keep breathing. That's the key. Breathe."
Join The Cahuamos Team


[Dec 17, 2011 3:00:40 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Hypernova
Master Cruncher
Audaces Fortuna Juvat ! Vaud - Switzerland
Joined: Dec 16, 2008
Post Count: 1908
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

The "Results Status" tab, all WUs tagged as "Validated" do mention how much has effectively been credited.
You can filter the results according to the tag, project or machine you want.
The other tags "In Progress, Pending Validation, Inconclusive, Error....etc" seem very clear and mean that you are note credited yet.
I am not sure that color coding brings much more except if filtering was not possible, and that all results were displayed completely mixed. But here it is not the case.
----------------------------------------

[Dec 18, 2011 1:58:58 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

Saw this yesterday [several times requested] and thought of the one weakness in this. The ever faster move off of validated result batches. You get barely time to blink with for instance Clean Water and a few other sciences, before they get transited to the Master DB, thus removed from the RS pages. Then What?

I'm running pirogue's WCGDAWS twice a day to capture everything. If a result changes status, a new timestamp is affixed and then sorting shows what has validated for the 12 hour period. This so far has not caused me to miss almost any validations. Some that take long to find quorum are moved off even quicker. Then you'll never know... but have full trust that credit was granted... The sideshow to doing science in distributed computing, which always lives center stage [how else do we know the TFL per CPU year contributed ;P ]

--//--
[Dec 18, 2011 2:42:09 PM]   Link   Report threatening or abusive post: please login first  Go to top 
gb009761
Master Cruncher
Scotland
Joined: Apr 6, 2005
Post Count: 2955
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

SekeRob, do any Validated WU's get removed from the DB before the next scheduled Stats update?

My guess*, is no - and thus, the best time to look would be just a few minutes past 00:00 & 12:00 UTC.

* As I say, this is a guess, but would like confirmation...
----------------------------------------

[Dec 18, 2011 3:30:45 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

Any 12 hour interval is fine it seems **, no guarentees. Seen on 5400 results in the WCGDAWS db only 3 misses... for ever sitting in PV and Inconclusive. New validations can disappear within a 12 hour period i.e. before the next stats run it thus seem, but it looks to be a rare coincidence and as noted, happens to those that took long to find a wingman. [or were stuck in "try validation" and noone did, then the servers concluding that 10 days was enough and transit them out as no additional wingman was needed ;>]

** Found 1 of the Beta's that completed first on Saturday, validated quickly and already gone.

--//--
[Dec 18, 2011 4:41:02 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

OK, I'll bite and put my 2p'th in too...

Unless I'm reading the results page wrong, the "Time Due / Return Time" column initially shows (in italics) when the WU is is due back but, once it is complete on your machine, it shows (in a normal font) when it was returned. For projects that are quorum 1 then it's likely that the result will be validated fairly quickly (unless there's an issue with the validator). But for projects with quorum>1 the result will not even be eligible to have the validator run against it until the quorum is reached. With resends that time delay could potentially be two weeks or more.

So I have some sympathy with CJSL 'cos sometimes WUs can sit in PV for ages and you don't really notice when they change to valid and so have been credited. Furthermore, if they disappear very quickly after validation, you might not even see them change before they're gone.

Personally (and in an ideal world, of course) I should like to see another column with "Validated Time" and I should like results to stay on the system for at least 84 hours after that time. I suggest 84 hours to allow you to check the results page only once per day, allowing for no checking at weekends, and allowing 12 hours leeway so that looking on Friday morning and Monday evening would be OK.

I did say 'in an ideal world', and I'm sure the techs have far more important things to do. But since the question was asked... devilish
[Dec 18, 2011 5:04:42 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

It used to be about 72 hours after validation, except then we did not run 700,000 results and more a day. Come 6 months, lest the techs are able to find more creative ways to elongate work as they've been doing to FAAH/HPF2 and seemingly too to GFAM, or combo multiple jobs in 1 task assigned, 1 million could become the norm and that has serious server performance implications, particular the scheduler that is the pivot of the whole operation and what you see on the RS pages. 84 hours is today utopic, compared to any standard BOINC grid that has results on for 24 hours if lucky. Batches completed belong in the scientists hands and not be lingering on the Result Status pages. [Don't ask about DDDT2... clueless why they are given the grand exception... maybe because there's just a few thousand in a run for A/B type :]

At any rate, the various ways of highlighting a validation in the current 24 hour WCG cycle, color coding, tickmarking or adding a 3rd timestamp to the send/received column has been discussed in past and think there was even a way decided on how to show it. If only there was time to do this and not cause a significant performance impact, the always present concern.

The sighted WCGDAWS tool does brilliant, so that's what quite a few trackers do and on top build a detail history and do analysis in many ways. I've got 5400 in there now and the programmer who made the tool has tens of thousands in... ready to filter, sort, average, print copy, slice and dice and export to a spreadsheet... and easily filter out the slackers for a RS page visit to see if it got stuck before it reached the validator.

--//--
[Dec 18, 2011 5:34:18 PM]   Link   Report threatening or abusive post: please login first  Go to top 
cjslman
Master Cruncher
Mexico
Joined: Nov 23, 2004
Post Count: 2082
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

Hypernova... Maybe I didn't make myself clear. When a WU is validated, that does mean that the results have been credited to my statistics. For example, if a WU is validated after the 12 hr cycle, it still hasn't been credited to my statistics (it will be credited after the following 12 hr cycle). My suggestion is to be able to distinguish between those validated WUs that have been credited to my statistics and those validated WU which still haven't been credited.
A couple of things....
- This would be really easy to do, because it's already implemented... it's just not being shown (which is my suggestion).
- I perfectly understand that this is not a high priority task and that the scientists probably have bigger and more important fish to fry.

CJSL
----------------------------------------
I follow the Gimli philosophy: "Keep breathing. That's the key. Breathe."
Join The Cahuamos Team


[Dec 19, 2011 1:51:26 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Hypernova
Master Cruncher
Audaces Fortuna Juvat ! Vaud - Switzerland
Joined: Dec 16, 2008
Post Count: 1908
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

Ok cjslman now I better understand what you meant. There is indeed a 12 hours period between statistics update and validated results.
But SekeRob comment is also valid. If Validated WUs are quickly pulled out then depending on the frequency at which you check your results, you would probably miss anyway the total picture.

Due to the huge number of results I have displayed in the Result Status list (around 5'000) I cannot start checking credits. I did once a time consuming statistic across some of my machines and for various projects on "credits requested" and "credits granted". The system is fair in average and that is what counts. Sometimes a WU gets 70% of the credits requested and another WU gets 120%. Understanding why is a specific WU has validated in a particular way is not really very relevant, except when there is a major problem like one project were a majority of WUs start erroring or having incredible runtimes etc. etc.

I have about 1'000 WUs that are validated every day so I use the Results Status in combination with Device Statistics function to check from anywhere if one of my rigs has stalled or has delays in sending WUs that could be a communication problem etc. etc. I can so identify which machine(s) has a trouble and when at home go in my basement and check.
----------------------------------------

[Dec 19, 2011 9:21:53 AM]   Link   Report threatening or abusive post: please login first  Go to top 
cjslman
Master Cruncher
Mexico
Joined: Nov 23, 2004
Post Count: 2082
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Indicate which validated WU have been credited

Not sure if I understood the following: "Validated WUs are quickly pulled out ..." ... hhhmmm ... I'm not suggesting for anything to be pulled out... My suggestion is just to have a visual indicator of which WUs have been credited to my statistics. I'm cool with the 12 hr cycle. What I don't know is the permanence/persistence of already validated/credited WUs on the Results Status page (and I'm not suggesting on changing that either). I'll try to give a simple sample:

I have two WUs which are showing up as valid:

CMD2_ 2201-1J2E_ A.clustersOccur-2A9U_ A.clustersOccur_ 17_ 1-- CJSLMAN Valid 12/18/11 10:47:28 12/19/11 09:35:54 6.00 68.4 / 75.4
CMD2_ 2201-1J2E_ A.clustersOccur-2H60_ A.clustersOccur_ 0_ 0-- spangler Valid 12/18/11 09:44:37 12/19/11 03:56:05 6.00 54.9 / 54.3

I have no idea if one or both or none have been credited to my statistics. Now if credited WUs were another color (say like blue) then I could know:

CMD2_ 2201-1J2E_ A.clustersOccur-2A9U_ A.clustersOccur_ 17_ 1-- CJSLMAN Valid 12/18/11 10:47:28 12/19/11 09:35:54 6.00 68.4 / 75.4
CMD2_ 2201-1J2E_ A.clustersOccur-2H60_ A.clustersOccur_ 0_ 0-- spangler Valid 12/18/11 09:44:37 12/19/11 03:56:05 6.00 54.9 / 54.3

... or maybe use a check mark or maybe change the status from "valid" to "Credit" or ... options are endless.

Again, this visual indication of credited WUs would be really easy to do, but I also understand that this suggestion would be a "nice to have" and that there are probably bigger issues to handle.

CJSL

CJSL
----------------------------------------
I follow the Gimli philosophy: "Keep breathing. That's the key. Breathe."
Join The Cahuamos Team


[Dec 19, 2011 1:48:46 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 15   Pages: 2   [ 1 2 | Next Page ]
[ Jump to Last Post ]
Post new Thread