foxfire,

With 20,000 results, my guess is that a result is being validated or removed from the database during the execution of your series of queries. This would change the results that are returned and would change the position of the results.

Can you add a check to your queries to see if "ResultsAvailable" changes during a set of queries?

If there is a change in that value during the iteration of your queries, then I suspect that is the root cause and we will have to think on what to do to help you.

Once we address this part, then we will take a look at the different data part of this since that one is odd and I wonder if it is somehow an artifact of of the first problem.

thanks,
Kevin

At a return rate of. e.g 1 per 8 seconds and a pull rate of 250 limit per second, that's the speed of return, there no hope in heck of ever getting a non-shifted series of fetches without incurring either doubles or missed when it involves 20000 with serverstate 5. In short such a series shows very often a different resultsavailable value from one 250 fetch to the next. That's the reason I've coded for 2 passes to catch the missed and eliminate any doubles and still the method is not watertight, which is why I'm coding for a different strategy now that the modtime filter function works. Be prepared for lots more hits from my end,... the shorter the results status retention, the more frequent. At 24000 a day I'll probably be pulling latest 2000 every hour, compare result name and mod time to then keep newest if the mod time changed or ditch any that are duplicate.

Would the API have included the unique db serial number, the number one sees when hovering a WU link on the results status pages, sorting and filtering could be lots more efficient.

Edit: The ending number in this link https://www.worldcommunitygrid.org/ms/device/....do?workunitId=1910194704 I. E. the workunitId which I'm guessing is the seed number for all copies in a distribution.

If the database id is unique and not reused it will solve my problems.
I'll post if the conflicting data happens again.
Thank you again for your help.

The finding is, not quite. Unique the workunitid may be, but see that for an MCM a different range is used as for FAHV which seem to be in the 1.18-1.19 billion numbering series such as 1189504048. More testing needed here, but the performance improvement is still expected to be significant.

Always 'assumed' [the standard pitfall] that the number we see at end of the WU link was one since start of WCG i.e. whilst we're racing towards 3.2 billion validated, the 1910194704 was the 1.9 billionth project task [then with it's suffixes indication the nth copy of that WU]. While it could be convenient to be able to link the API Id number to the one seen on the RS pages, it's at least to me of no real use, any permanently unique and incrementing number for my archive is as good as any (unawares there was a second series).

The 'narrowing down' method in tests can increase according speed of finding a match by a factor 1000, yes 1 thousand [the more records, the bigger the exponentiation], and currently by far the biggest bog down when recomputing in full a workbook that eats 1.4Gb RAM. Add to that the ModTime method, each hour just and -only- fetching anything changed/added in the past 3600 seconds + little extra to cover the serial fetches time [1x250 per second, 2000 needing 8 seconds], is bound to improve the execution times massively (largest live archive tested was 410,000 results). Now it's a case of applying the necessary changes to the code and robustifying. The end goal, not missing a single 'credited' result.

thx for your swift action :)

P.S. "The id of the result which is one of the copies of the workunit" is probably better as when you happen to get 2 or more copies spread over multiple machines for the same member [the bigger the farm, the greater the chance], the WU number could cause a duplicate index number, i.e. that key could not be set to unique, whilst it does make identifying such a situation easier. A rarity but for the few.