Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go ยป
No member browsing this thread
Thread Status: Active
Total posts in this thread: 127
Posts: 127   Pages: 13   [ Previous Page | 1 2 3 4 5 6 7 8 9 10 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 14658 times and has 126 replies Next Thread
uplinger
Former World Community Grid Tech
Joined: May 23, 2005
Post Count: 3952
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

Again, we are sorry for the issues with uploading to our servers. We are still working at getting this sorted out.

Thanks for your patience,
-Uplinger
[Apr 7, 2014 5:21:10 AM]   Link   Report threatening or abusive post: please login first  Go to top 
keithhenry
Ace Cruncher
Senile old farts of the world ....uh.....uh..... nevermind
Joined: Nov 18, 2004
Post Count: 18665
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

Again, we are sorry for the issues with uploading to our servers. We are still working at getting this sorted out.

Thanks for your patience,
-Uplinger


You've had a VERY long day after doing the maintenance the night before. Thanks for all the hard work and make sure you get to take Friday as a comp day so you can enjoy a three-day weekend. With uploads enabled (for now at least), know we're hoping the upload server deals with the flood! You've had enough of this. biggrin Would having the SFS folks add more space help any? Do we have just the one upload server? Would having a second (or another) one (even though it's probably virtual) with its own disk space offer options for allocating projects to each and spread the load on the disk?
----------------------------------------
Join/Website/IMODB



[Apr 7, 2014 5:33:41 AM]   Link   Report threatening or abusive post: please login first  Go to top 
uplinger
Former World Community Grid Tech
Joined: May 23, 2005
Post Count: 3952
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

keithhenry,

Thank you for your kind words.

What I'm seeing is that there is a bottle neck with mcm1 assimilation. I am trying to speed things up with that. I have been able to clear 10% of the backlog on it in the past few hours, so I'm hoping a few more hours will allow the backlog to clear out a bit more. Adding more space would just be a temporary fix, but we are working towards a permanent one.

I have however wrote a script that will disable uploads automatically when storage gets within 25GB of being full. This will happen automatically throughout the night. The reason I have it set to 25GB is that new work getting loaded and indexed goes into the same storage, I need that space to make sure work keeps flowing and members can still download work. It is set to allow uploads when it gets above 50GB of space, so there will be periods of no uploads, then uploads....

I will be running more tests in the morning to see if I can optimize the assimilation even more, but at this point, I would like to see if the changes I have made will catch up on the backlog.

Thanks,
-Uplinger
[Apr 7, 2014 6:08:06 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

Understood Keith, have a nice rest wink
[Apr 7, 2014 6:14:23 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

Thanks uplinger, your hard work is really appreciated. biggrin biggrin
[Apr 7, 2014 7:59:17 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Former Member
Cruncher
Joined: May 22, 2018
Post Count: 0
Status: Offline
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

I'd like to add my thanks to Keith too. It's been a very long weekend for you and you've done a great job of trying to keep everything flowing. I hope you get some down-time to compensate.
[Apr 7, 2014 9:15:22 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Movieman
Veteran Cruncher
Joined: Sep 9, 2006
Post Count: 1042
Status: Offline
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

Sorry, I was hopeful that the file system would stay below a certain level while I got some rest. My hopes did not come through for me. The root cause is that MCM1 has files that are larger than average being returned right now. This has caused the file system to fill up. There were some cleanup I could do after the maintenance outage, unfortunately that was only a temporary fix. I have take a few other steps on the backend to potentially free up some space as well.

Thanks,
-Uplinger


I have an old 6.4 gig WD HD I can send if that will help.. Heck, maybe even a 40 gig! biggrin
----------------------------------------

[Apr 7, 2014 10:08:28 AM]   Link   Report threatening or abusive post: please login first  Go to top 
robertmiles
Senior Cruncher
US
Joined: Apr 16, 2008
Post Count: 443
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

boinc says upload pending project backoff
Go to Transfer tab and hit retry biggrin


I just tried that and the uploads failed again. The log file says maintenance underway on the server.
[Apr 7, 2014 12:02:17 PM]   Link   Report threatening or abusive post: please login first  Go to top 
robertmiles
Senior Cruncher
US
Joined: Apr 16, 2008
Post Count: 443
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

keithhenry,

Thank you for your kind words.

What I'm seeing is that there is a bottle neck with mcm1 assimilation. I am trying to speed things up with that. I have been able to clear 10% of the backlog on it in the past few hours, so I'm hoping a few more hours will allow the backlog to clear out a bit more. Adding more space would just be a temporary fix, but we are working towards a permanent one.

I have however wrote a script that will disable uploads automatically when storage gets within 25GB of being full. This will happen automatically throughout the night. The reason I have it set to 25GB is that new work getting loaded and indexed goes into the same storage, I need that space to make sure work keeps flowing and members can still download work. It is set to allow uploads when it gets above 50GB of space, so there will be periods of no uploads, then uploads....

I will be running more tests in the morning to see if I can optimize the assimilation even more, but at this point, I would like to see if the changes I have made will catch up on the backlog.

Thanks,
-Uplinger


An idea to consider: Can you set a disk space limit at which mcm1 uploads and downloads are blocked, and a higher limit at which all uploads and downloads are blocked?
[Apr 7, 2014 12:25:24 PM]   Link   Report threatening or abusive post: please login first  Go to top 
uplinger
Former World Community Grid Tech
Joined: May 23, 2005
Post Count: 3952
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Error reported by file upload server: No space left on server

Robert,

Thanks for the suggestion, unfortunately we can not do something that blocks uploads on an application basis.

-Uplinger
[Apr 7, 2014 12:35:20 PM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 127   Pages: 13   [ Previous Page | 1 2 3 4 5 6 7 8 9 10 | Next Page ]
[ Jump to Last Post ]
Post new Thread