||12-11-2006 09:25 PM
What happened to WF?!?!?!
Good evening folks,
Ok so im sure everyone wants to know exactley what happened, so may not care and are just glad that we are back online, others may even still be upset that we are online at all.
2 nights ago I was sitting at my desk at work and decided to do my usual morning check of wf. upon trying to reach the site I was met with a blank page. This is nothing new neccesarily so I shrugged it off and checked back in about an hour. When I still couldnt reach it, I called my wife and woke her up, and had her log on from home. Bri says " cant get on wf, now leave me alone im going back to sleep"
This prompts a call to the datacenter that houses our dedicated server, upon reaching them I get a great customer service rep that tells me he will put in a reboot order for me.
A few mins later, the forum is up, and I check my trouble ticket and there is a simple line "rebooting" and I think GREAT! it worked. 5 mins later, server down again, recheck trouble ticket and see a quick explanation of "consoling into server to investigate something"
at this point I think to myself WTF are you investigating, this is my production server smacky, push the button like the reboot monkey you are and stay away from my stuff. But I dont write this because having been in IT for a while I know that more IT folks think they are more important than they are and tend to try and go above and beyond and be in control situations. So I add to ticket, Bring server up, no need to console into server.
server stays down, all of a sudden there is an unknown "issue" that wont let my server come up. I call BS. this prompts many angry phone calls, insistance on a different tech doing my bootup while im on the phone which is in my service agreement (remote hand and eyes) however if the remote hand and eyes arent giving you all of the information you cant make good calls. So I call a rep that I know at the company and request he go get my server online. He calls me back and tells me he isnt sure what is wrong but there has been partial corruption on harddrive and that he can get it back online in limited capacity. essentially I gained access to look at the filesystem and ftp in. which is fine except you cant make Database backups in that fashion for the most part. So I had to replicate everything off the server to a home server and do it all that way, as the tech borked most useful software on the server
Now why did this take me 48 hrs? well I work a 4 on 4 off shift 12 hr days . I spend 3 hrs a day commuting, so outside of work, i have just enough time to sleep. Kyle didnt do the restore because he did not have access to the tools I do and this was basically a 1 shot deal as our server is best used as a paperweight at this point. We are back up and running on a hosting account temporarily until we procure a new server and colocation facility where I can go and physically touch my server rather than have a low wage tech screw our stuff up in the future.
Thankfully I was able to pull an up to the minute dump of everything. I did get an error on rebuilding everything that I am unsure of so If you find something broken PLEASE pm me so I can address it... All posts and users appear intact however
Thanks and I hope we dont lose too much of our userbase over this mutliday downtime. Thanks for your support guys.