Web Server Upgrade, Part 6

Finally!

After all the trouble and hassle with the last host, I think I've finally found a permanent home for the infopackets.com web site and virtual-host friends who are sharing this dedicated server.

A brief, but very interesting (hellish) synopsis entails:

Some time near the end of October 2002: It was decided that I needed to get another dedicated server to handle all the traffic that comes through the infopackets web server.

October 30, 2002: After being reassured by a "knowledgeable" web-hosting sales representative that infopackets.com and its high demand for bandwidth could be properly managed, I agreed to have the web site handled by a new web hosting company -- herein, referred to as Company A.

November 1 ~ 14, 2002: Problems of sluggish server response and high packet loss are reported to Company A through their online trouble ticket system and ongoing phone calls to tech support. Reported problems are relatively minor; at this point, there has not been 2 consecutive days without some sort of issue with the server.

November 15 ~ 18: An email is received from Company A which outlines that there has been "some" downtime during the last 72 hours while their networking technicians reroute the network for an upgrade. At this same time, Company A decides to implement a network switch and limit bandwidth to and from the infopackets web server.

November 19 ~ 21: Infopackets.com suffers from an unacceptable ping response of over 3,000 ms (average ping response are usually 25 ~ 100 ms). A weary network representative from Company A reports that infopackets.com is one of their top 10 bandwidth hogs, and that capping infopackets.com at the switch is necessary or he will lose his job because the "bandwidth bill for our Company this month is sky high."

No kidding.

On another note, a computer programmer from Company A makes a very cool [apache web server throttling] program which helps to flow bandwidth evenly to and from the infopackets web server. For the time being, the program eliminates the extremely unacceptable 3,000 ms ping response.

November 22 ~ 23: Networking problems persist. Extremely high ping response times and server timeouts still plague the infopackets web server, even after the very cool apache throttling program is implemented. At this point, there has not been 2 consecutive days without some sort of issue with the server.

Am I repeating myself?

After approximately 23 days of pure hell, a decision is made to dump Company A and their web hosting and find one that is competent and can handle relatively high bandwidth consumption without "dogging the server" during peak times.

November 24: A new server is purchased with Company B. Files from the infopackets web site and virtual hosts are moved over to Company B's web server. Nameserver records are updated and set to propagate at this time, which should complete in approximately 72 hours. Company A is informed to cancel the infopackets web server at the end of the 30 day money-back guarantee period.

November 25: The billing representative from Company A neglected to read in the letter of cancellation that time was needed to propagate the infopackets web server to its new host, and a decision is made to axe the server effective immediately. Service is restored 3 hours later when it is discovered that the web server had been disassembled.

November 28: An email with regard to service cancellation and refund is received. The choices are:

• disband immediately; receive partial refund
• stay online for 1 extra day, pay for 1 month of hosting
• stay online for 2 days, pay for 2 months of hosting

This is stated in an email from the billing representative of Company A:

" After reviewing the situation I have decided the best solution is to give you a choice. You have had 72 hours [actually, less than 24 hours] to transfer your domains to your new server and we are not - as you suggested - obligated to give you 30 days free service.

In short, although you did have some difficulties, I feel you are now using us unfairly*. Consequently, we will cancel your service this afternoon and you will be refunded all but your setup fee. Optionally, you can request to stay on till the 30th [1.5 days from now] and receive no refund, or you can keep your server until after December 1st and pay for another month's service [in total, pay for 2 months of service in a period of 2 days time]. "

I emailed back and decide to "bail" and pray that the nameservers will propagate in time. 5 minutes later, Company A disconnected the server.

* I wouldn't be in this situation in the first place if I was properly informed about what service Company A could provide. I had a representative from Company A on the phone and had him compare my previous web host to Company A's capabilities, and he assured me to "the best of his knowledge" that I "shouldn't have a problem with high bandwidth issues" and that moving over to company A would be the right choice.

The fact is that Company A's network was out of control for a 2 week span because their network wasn't properly configured (broken switch?) and had my site "uncapped" at a 10 megabit connection which inevitably sucked back a lot of bandwidth.

The billing representative told me on the phone that he thought it was fair that I should "help pay" for their unwieldy network usage (and thus, pay the initial setup fee). The truth is that I was not aware that the server was running @ 10 megabits until I was told by one of their networking representatives at same time they decided to cap the server at their switch and royally screw up the server response.

November 29: After receiving almost no traffic to the site after 3 days, I check the nameserver records and discover that the Authorative Nameservers were not set. I initialize the Authorative Nameservers and propagation to the new server and traffic begins to trickle through the next day.

December 01: Satisfied that everything is configured properly and all the hosts have been moved over, I decide to clean up the server by removing temporary files left over from the move. The temporary files were located in folder in my "home" directory, which also had all of the active infopackets web server files.

The unix command entered was:

rm -R -f backup/ *

In Unix, the rm -R -f recursively deletes all directories which are located in the last argument. If you pay close attention to the above line, you'll notice that there is a space after the forward slash, and before the *.

I wanted to cry.

Instead of deleting the backup directory, the command above deleted the entire infopackets web server directory (virtual hosts were still in tact). I immediately began to freak out and wanted to put my head through the monitor, but I remained relatively calm and called Company B for assistance in retrieving deleted files.

Could anything else bad happen?

Alas, I was greeted with more bad news.

A voice from Company B's tech support said, "Sorry, sir, but the rm -R -f command is pretty much permanent. You can try a 3rd party recovery utility, but we don't support that. If you use it and it screws up, we're not responsible."

In a blink of an eye, all 200 of the newly subscribed Infopackets Readers were gone.

I attempted the recovery program but it didn't work, and decided to restore a backup of the infopackets web site from an archived file. Unfortunately, I lost over 200 subscribers that managed to sign up for the newsletter in the past 24 hours.

Final thoughts

My experience with Company A was a nightmare.

Since I've moved over to web hosting Company B, there hasn't been a problem with the server yet... and it's screaming fast. In all, I'm paying roughly the same price for Company A's service, but I'm getting 2x more storage and 6.66x more bandwidth capability at any given moment.