Hope you get the answer you are looking for

TLDR summary: These are estimates. Estimates always have error in them. The technique we use is reasonable given the data available to us but it will make mistakes from time to time.

Longer version:

There is a continuing assumption that we know how much work a given project is going to have at the start of the project. That is a BAD ASSUMPTION. We do not know how long each project will be. At the start of a project the researchers have a proposed scope that they want to run. This usually means that have a number of data-sets in mind at the start of the project that they want to run through the grid. As the project runs and the researchers look at the data they get back, they usually have some new ideas and add some new data-sets or remove others. Thus the total number of data-sets run usually winds up being different, sometimes significantly so, then what was expected at the start of the project.

Each of these data-sets usually gets converted into some number of batches. Each of those batches winds up having an unknown number of workunits. The reason that it is unknown is that the run-time length of the work in a batch is often dependent on the particular data being run. We break those batches up into a larger or smaller number of workunits based on how "tough" that particular data is to run. We are also not usually able to asses how tough a "batch" is until we have downloaded it onto our servers to prepare for running on the grid.

Between these two factors, it is very hard to asses the total amount of work that will be run for a given project. Thus we have an estimate based on the information that we have available.

In addition, the rate that we are working through a project depends on a number of factors. How many users are currently participating? How many projects do we have currently running? Does a team, organization or corporate partner decide to focus their resources on a specific research project instead of everything that we are currently running? Each of these items changes the rate at which work is being run for a particular project. As a result, we take a recent average of the pace and use that as an estimate.

This means that the % complete for a project is:

estimate of recent work done for the project/current estimate of work remaining for the project

Since both of those change and are estimates with a some uncertainty in each measurement, the end result is going to vary and change.

We do want to change how we present the estimate. Using a number like 91% provide a false precision in the estimate which is why It think that there is often confusion and frustration about the number. It would be more accurate and clear for us to provide a metric/visualization that simply said one of the following:
- The project just started
- The project is early in its life-cycle
- The project is in the middle of its life-cycle
- The project is late in its life-cycle
- The project is ending soon (within 2 months)

That is all the precision that we can reasonably provide and I think it would be good for us to make a change so that the various webpages reflect that.

As for when we will get to this update. There always seems to be about 100 things that really really need to get done so I don't want to estimate when a change like this might be made.

Thank you Kevin for your detailed explanation. I too , like Mumak think just having the levels you mentioned would be a shame. I understand that the current process is rife with multiplying uncertainties at all the various levels. However, I believe that when you (meaning the IBM team) vet a project, one of the items always considered is the estimated scope of the project. That scope may be expressed in potential work units, possibly in time or maybe in terms of how much information can be processed until the money runs out. I understand the potential scope is riddled with uncertainty, both by the scientists and your team, but it would still be somewhat helpful and informative if we (the crunchers) had a rough idea of the size and scope of the project(s). Furthermore it would also be nice to be informed if the scope of the project changes significantly. Just a few sentences such as: The scientists added some work and doubled the size of their project. Or: The existing work done to this point has shown little benefit and the scientists are abandoning this approach to pursue more promising leads at a future date thus cutting this project short of its intended run.
I do agree the percentage approach lends a degree of precision which is not justified. Perhaps the percentages could be expressed as a range i.e 10% to 20% completed using a decile approach.
Keep up the good work.
Cheers

Murnak said:

^ This, to a tee. This is exactly the KISS calculation that should be done that doesn't need to unnecessarily introduce a time or rate variable into the percent complete calculation. Simply [# of batches or work units processed] / [# of known batches or work units total] * 100%. If we see the number go down, it's because the denominator increased (i.e., more work or batches were added to the project).

I believe it muddies the waters to introduce rate or time variables into a percent complete calculation. Just keep it simple.

I think that at least the approximate number of total batches is known from the beginning. Sure, it can change, but I don't think that happens too often and even if, we're OK with that. If that would happen, we'd notice it immediately on the percentage completed and know why it happened. Forget any performance estimations, we're aiming for a simple indicator. Of course this won't work for a precise ECD estimation, which would also require to include the current throughput in calculation. Such estimation could be left to your thread ;-)

As for SCC, I believe that change in throughput is that the units run quicker now. A month ago they took ~2h, now ~1h.