Never Patch Your Virtual Server

 wayne on August 12, 2008 

Here's how a well intentioned email goes wrong.  Horribly wrong.  I'm not naming names and pointing fingers, however the biggest finger needs to be pointed at myself.  I misread a page that I was reading.  The email gave me the link, I clicked it, and put into motion a series of events that left me, well, mildly calm.

The Email

The email was from my host for the virtual server that I run this site on.  The email was stating that there is a DNS poisoning thing going around affecting DNS servers, which I am also running on the same box.  A bit outside my area of expertise really, but that's why it's on there, so I could learn it. 

I had everything wired through the DNS system.  For example, when I want to check my email from the web, I'd go to a particular URL like mail.google.com.  Well, because my I'm running my own DNS, I slowly lost my connection to these services.  I could move the settings to another server, but changes like that take time to propagate around the Internet, and I thought I would only be in this pickle for a little bit.

Anyway, there was a link in the email that I clicked to find out more information about the potential attack, and I read the article.  In the article, I mistook one action over another that basically said:

Ok, so I misread the email and proceeded to log in and check out Windows Update.  I then installed everything there was to install, and there was quite a bit.  However, I didn't check the install for Internet Explorer 7.  I don't need a browser on the box anyway, so why bother.  That has nothing to do with anything really, I just thought I would point that out.

Once the update was complete, I'm feeling pretty good about things.  All the while I'm writing a quick post, doing this and that and I finally end up at the prompt 'Do You Want to Reboot now, or later?".   I begin my reboot.

I figured it may take about a minute or two before I get my server back, but after those few minutes pass, I start to get this sinking feeling.  What if it doesn't come back?  Oh crap, I don't have a current backup! 

I still had a few posts that I had pre-published in the queue, hence why 3 popped out at each subscriber when the site came back up.

The Support

After a few minutes I logged on to my account interface and checked out the web stats on my server.  Mind you, the servers down, I'm now reaching into my vendors account to see how they see the status of my server.  Hmm...no sign anywhere that my server is not coming back.  That's fine, the tools they provide just don't show it, I'm gonna have to call in to get this corrected. 

At this point, the site has been dark for about an hour.  I call support, and because when I was in the admin interface and had specified for the system to do a complete recycle of my virtual system, I would have to wait for it to come around in the queue.  That wait lasted way too long. 

I was told the system will reboot my virtual server within 2 hours, according to the engineers there.  Well, actually I think I may have ruined their day with the mess I realize (at this point) I had made.  I really screwed the pooch.

After waiting for a few hours and still no sign of my site or access to the virtual server, I call in again and express my concern.  This time I get 'It could be up to 4 hours...'.  I decide to call it a night, and check it when I wake.

The next morning the birds were chirping, the morning air smelled great, my mind started wandering....Oh crap!  Instantly it hit me, my damn site is down...has been for all night.  I need to get this thing back up and running. 

I'm going to cut to the chase here and say that the entire day was spent actually getting some solid work done and really checking in with support to see what can be done.  The site was still dark. I can't access it in any way shape or form. I can't get my email at all.  I think to myself 'I could just transfer it to a new instance of a virtual server...', but the server may come up soon.  Oh decisions...leave me alone.

I went back to work.  I'm at the mercy of the support team on the other end of the phone to get my box back up and running.  I finally get a very nice person named Nancy whom stuck with me on the phone while she pressed hard for answers to the issue while I stayed on hold.  I think the call lasted around 2 hours total.  There were a couple call-backs and all the while it is now Sunday night. 

Weren't we just talking about Saturday?  Yeah, we were, but see how it feels?  Some weekends are like that, I'm able to sit in my office and simply bust out all kinds of sexy looking code, and time will just fly right by me.  One of the benefits to keeping things simple, I guess.  Sometimes we party on weekends, other times I'm in the man cave and only occasionally poke my head out.

This particular weekend saw the most use of my cell ear-piece than any other.  People that happened to be in the house would occasionally see me floating by talking with myself.  Other times they wondered if I was on the phone or just forgot to de-suction the thing from my head. 

I made a few calls over the weekend to some cool people too, so it wasn't all business.  I also managed to make it in the pool a few times.  It sometimes gets hot in the office and sitting all day will really cramp your style from time to time.  It's nice to be able to pop outside and float for a few.  Maybe raise the blood pressure a bit...but not this time.  I was relaxing this weekend.  It was like I had walked out of the room that is the Internet and turned off the light behind me.  Hell, with that simile you might say I did something green.  Sure.

Anyway, I'm side tracking a little, but this girl Nancy really held up her end well and provided some top-notch support.  She gained me access to the server so that I could finally...FINALLY get a backup of my files.

In the end, I asked them to redo my server from scratch so I can start fresh.  Thank you Nancy!

The following morning, Monday, I spend that evening getting everything back to normal.  Installing the components I need on the server, moving the site files, setting up the DNS server again, because I still have not moved it off there.  I battled with this, and I decided to continue doing my own DNS for now.  I can flip it over to the vendor later if I need to.  I'll just make provisions to replicate my settings on another server that can be used as a failover or secondary DNS server.

Long and short of it (too late) I had my site back up and running.  A few bug fixes later, I'm back to normal operations, and here is the first post on the new server. 

What did I learn?

Well, a few things.  I was ever so humbled by the fact that Gary had just went through a template loss.  Like I said to him before, I know how you might feel when that happens.  This just happened to be quite an exaggeration of that, so much so that it's laughable.  The entire weekend I was quite serious, but I never let things get to me and was a pretty cool cat through the weekend.  However, I was secretly pulling my hair out because I had no power to control the situation at all.  I was at the mercy of a corporation, and I really despise that.

Here's what I learned:

1. A neat trick to talk your co-workers into doing?  No, a great tragedy for my peers to laugh with/at me.  You know they will when they find out, why try to dodge it...
2. Can I get a big amen to having a back-up, or being able to grab a back-up?  Thank you brothers and sisters.
3. A lost feeling can overtake the senses, causing one to drink beer and go swimming.  Odd, I never noticed that's how it happens, but by gosh that's how it happens!  WOO HOO!
4. Always, always make sure you read what's in front of you.  It may help you.  Perhaps slightly, but it's still help right?
5. It was nice to completely detach for a few days.  I did things I don't normally do, like stand up, walk...things of that nature.
6. The phone is a replacement for when the Internet is down.  You would think I turned Chatty Cathy I talked to so many different people....
7. Being dark isn't fun when it's your web site you're talking about.
8. I need to change the way I save drafts.  I need to save them locally as opposed to on the site in some far off future date that I pull from.  :)
9. Never write a post about your site rebooting.  It is now considered bad mojo, and I won't have any part of it anymore.
10. It really sucks when your site goes dark.

We shall now resume our regularly scheduled postings.

Permalink or Trackback Comments (8) General IT
Technorati: No reaction yet!

Tags: hosting, issues
Actions: E-mail

Was this helpful?

If you liked this or found it helpful, please digg it, stumble it, buzz it, whatever it, to say thank you.

 

 If you would like to receive these posts as they happen, you can subscribe to my feed or receive my posts in your email.