Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go ยป
No member browsing this thread
Thread Status: Active
Total posts in this thread: 88
Posts: 88   Pages: 9   [ Previous Page | 1 2 3 4 5 6 7 8 9 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 8347 times and has 87 replies Next Thread
bfmorse
Senior Cruncher
US
Joined: Jul 26, 2009
Post Count: 296
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

My VALID, MCM WU's that are still vislble:

My oldest VALID WU has a return time of "2023-10-24 01:13:34 UTC".

I still show 35326 items in the current listing down from the initial 36038 items from my posting on Nov 10, 2023 2:23:17 PM.

New WU's are still slow to me. So, I have not yet re-energized those systems previously powered off. My queue is typically set for zero additional days as i tend to get a lot (relatively speaking) of resends.
[Nov 13, 2023 8:22:39 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Speedy51
Veteran Cruncher
New Zealand
Joined: Nov 4, 2005
Post Count: 1288
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

I have over 1700 valid results but I don't see any issue with this. I process about 23 an hour
----------------------------------------

[Nov 14, 2023 1:23:51 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2155
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

I have over 1700 valid results but I don't see any issue with this. I process about 23 an hour

So, Speedy51, when you open your Results page and you go to the last page (in your case that would be 1700 divided by 25 - if you have 25 items per page - which equals 68) of all your selected Valid results, you would see nothing out of the ordinary confused

My oldest Valid result (currently from page 669) is:
Result name         Status Sent time           Due / Return time   CPUtime/Elapsed Claimed/Granted
MCM1_0206648_5472_0 Valid 2023-10-23 23:42:31 2023-10-24 14:09:51 2.52/2.54 75.4/75

Adri
----------------------------------------
[Edit 1 times, last edit by adriverhoef at Nov 14, 2023 12:47:45 PM]
[Nov 14, 2023 12:44:34 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Mike.Gibson
Ace Cruncher
England
Joined: Aug 23, 2007
Post Count: 12359
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

My oldest is:
MCM1_0206667_9530_1 Mike-PC3 Valid 2023-10-23 23:00:24 UTC 2023-10-29 23:00:24 UTC 2023-10-24 06:21:49 UTC 2.35 / 2.36 61.1 / 67.6

Same date as Adri but a bit earlier in the day.

Mike
[Nov 14, 2023 2:48:22 PM]   Link   Report threatening or abusive post: please login first  Go to top 
alanb1951
Veteran Cruncher
Joined: Jan 20, 2006
Post Count: 952
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

Further to Adri and Mike's observations:

My oldest is MCM1_0206667_9817_0 (WU 407998880) -- it was returned on 2023-10-24 at 02:42:57 (UTC)

If the data in the ModTime field returned by the old API can be believed, it was validated at 02:43:16 the same day.

I see no sign that assimilation is happening at present, so there's an associated worry. According to the Global Statistics History MCM1 is seeing over a million returned (and credited) results a day at present, so that's over 500,000 workunits a day. With 20 or more days where almost no WUs have been assimilated and purged, that's an enormous backlog to clear! A standard assimilator does up to 1000 WUs at a time (but how long that might take is an unknown)... Given how far it got clearing out stuff a few days ago, I suspect it might take a week or more of uninterrupted running! I leave any further calculations regarding this as an exercise for the reader :-)

Cheers - Al
[Nov 14, 2023 9:03:09 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Unixchick
Veteran Cruncher
Joined: Apr 16, 2020
Post Count: 949
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

The best we can do is to keep screaming on the boards that this is an issue. I've seen this on numerous BOINC sites. When the database gets too large, it all just grinds to a halt.

I take comfort in that there is a group of us seeing the same thing. I do like to see the data. The next piece of data is how long can it run like this before it stops??
[Nov 15, 2023 3:34:40 PM]   Link   Report threatening or abusive post: please login first  Go to top 
alanb1951
Veteran Cruncher
Joined: Jan 20, 2006
Post Count: 952
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

Unixchick ,
The best we can do is to keep screaming on the boards that this is an issue. I've seen this on numerous BOINC sites. When the database gets too large, it all just grinds to a halt.
Yup -- that's one of the possibilities I considered as "requires drastic action" in my post in the Chat Room thread on 14th November. And it is having an effect -- collecting or refereshing statistics that come direct from the BOINC database can be quite sluggish (whether via API or web pages...)
I take comfort in that there is a group of us seeing the same thing. I do like to see the data. The next piece of data is how long can it run like this before it stops??
The "Statistics History" for MCM1 reports nearly 30 million results returned since 24th October (inclusive of that day); that suggests somewhere in the vicinity of 15 million WUs stuck waiting for assimilation! That's a lot of items to scan looking for WUs ready for assimilation[*1].

Unfortunately, the solution is likely to involve either shutting off the work-flow to and from users (to avoid further build-up) or effectively taking the whole service off-line for long enough to do something (such as multiple MCM1 assimilators if not already in use?) to clear some of the backlog[*2] with (hopefully) reduced stress on the database. For all we know, the techs may already be juggling various "scheduler" processes in an effort to sort things out, but there doesn't seem to be much progress :-(

Cheers - Al.

*1 The much smaller number of OPNG tasks live on the database at any time potentially makes for a much quicker scan -- it appears that OPNG tasks still clear within a couple of days of validation, so the database can still cope at present if circumstances are favourable!

*2 However, imagine the outcry from a fair proportion of the user base if anything interrupts work flow, even if it's an essential action! (Totally unlike the ARP1 situation, where it's obvious that the blame should not be laid on WCG...)

[Edited regarding sluggish statistical data delivery, and on "blame"]
----------------------------------------
[Edit 2 times, last edit by alanb1951 at Nov 15, 2023 6:50:49 PM]
[Nov 15, 2023 6:39:22 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TigerLily
Senior Cruncher
Joined: May 26, 2023
Post Count: 280
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

As many of you have noticed, we are having an issue with purging MCM1 workunits and recently we applied a fix that we believed would also address the symptom, dead MCM1 assimilators that would quit again once restarted. We are investigating why the MCM1 assimilator services are failing again now, and will update everyone when we are able to bring the MCM1 assimilators back up in a useful state.

Regarding the results backlog, we should be able to cope with it once we can get the MCM1 assimilators back up and running.
[Nov 15, 2023 8:09:14 PM]   Link   Report threatening or abusive post: please login first  Go to top 
alanb1951
Veteran Cruncher
Joined: Jan 20, 2006
Post Count: 952
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

TigerLily,

Thanks for the confirmation of the issue and the efforts to resolve it.

I have visions of many megabytes of assimilator log files and the accompanying headaches...

Cheers - Al.
----------------------------------------
[Edit 2 times, last edit by alanb1951 at Nov 15, 2023 10:28:27 PM]
[Nov 15, 2023 10:26:29 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Unixchick
Veteran Cruncher
Joined: Apr 16, 2020
Post Count: 949
Status: Recently Active
Project Badges:
Reply to this Post  Reply with Quote 
Re: Validated but not purged

Thank you TigerLily. It always makes me feel better to know that the problem is acknowledged and recognized.
[Nov 15, 2023 11:56:13 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 88   Pages: 9   [ Previous Page | 1 2 3 4 5 6 7 8 9 | Next Page ]
[ Jump to Last Post ]
Post new Thread