The Mapping Cancer Markers project has result files that are somewhat larger than our normal projects. As a result, they are taking up significant storage on our servers. While we have taken a number of steps to free up space we have come to the conclusion that we need to change how we run the project in the short term while we come up with a long term solution.

We exchange data with the researchers in "batches". The researchers produce a set of related tasks as a batch and we download that batch, prepare it for running on the grid, load it, collect all the results for the batch and then transfer the completed batch back to the researchers.

The majority of results for a batch (>80%) are received in the first three days from when the first task is distributed for the batch. However, due to the 10 day deadline and time for 2 resends, it takes about 16 days for a batch to complete. This means that we have to temporarily store that 80% of a batch for an extra 13 days.

By reducing the deadline for a job to 7 days from 10 days, and increase the time allowed for resends to 35% of the original time, we are able to reduce the time to complete a batch in about 12 days. This will reduce the time that in-progress batches are stored on our system by about 4 days and therefore reduce the required storage.

We are working on what the best longer term plan is, but this step is necessary in the short term. You will start seeing the shorter deadlines in the next 24 hours.