Roadrunner Nightmare

Roadrunner Nightmare - Resolved

Around 5pm on December 30, the trouble that I now understand affected every Hawaii ISP that uses UUNet - including MSN, Verizon, Lava.net, Aloha.net, Turquoise.net, and many more - was cleared.

However, additional issues remain affecting Roadrunner Hawaii users.

Aaron Stene, a friend and another local Roadrunner and Turquoise.net customer, got through to Jason Rounds, a UUNet Customer Liaison:

From: "UUNET Customer Liaison" <help@UU.NET>
To: "Aaron Stene" 
Sent: Saturday, December 30, 2000 17:05
Subject: Re: Palo Alto POP High Latency (UU2406337)

Thanks for your EXCELLENT trace route info, it has helped greatly.  I had
the operations Center check the CR device and it was locking up, they
cleared the routes and it seems to be working ok now.

Since at least November, and possibly as early as Rob Perez's October Star-Bulletin article on the 'Hawaii Bandwidth Crunch', this problem at MCI/Worldcom's Palo-Alto, California UUNet facility has been a big part of what really isn't a bandwidth crunch at all! It appears that UUNet and all of its customers never even identified the problem. Aaron and I were bombarding UUNet and Roadrunner with e-mails for more than a week, and didn't stop until somebody finally did something!

It is truly outrageous that this situation has affected thousands of Hawaii users for months and the people that should have identified and corrected it never did.

Compare the trace on the first page to Zonelabs to the one I get now - there is not a single hop that shows a latency spike:

I sent the following e-mail to Kiman Wong, General Manager of Roadrunner/Oceanic Internet on 31-Dec-00:

Kiman,

Around 5pm yesterday, the problem with RR/UUNet cleared. I haven't heard a thing from anyone at Oceanic/RR, and it appears that this problem was cleared as the result of Aaron Stene's contact with Jason Rounds at UUNet. Both he and I have been bombarding UUNet with e-mails and phone calls regarding the problem that UUNet customers like Roadrunner are supposed to deal with. (For the most part, UUNet ignores end users and that's what we are.)

It also appears to me that this problem has been going on for way longer than the 2 weeks RR changed routing for West Hawaii from AT&T to UUNet: I believe the UUNet routing problem affected other RR customers assigned to UUNet since at least November, and also affected customers of every ISP in Hawaii that uses UUNet. (Including MSN, Verizon/GTE, LavaNet, Aloha.net and many more.) In fact, I think this problem existed at the time of Mr. Perez's October article in the Star Bulletin regarding the Hawaii 'bandwidth crunch', and that a big part of this crunch has been faulty UUNet routing. (Also see: https://modemsite.com/56k/clogged.htm which I wrote on 18-Nov-00: "Friends are telling me it takes 5-10 minutes to bring up a single page with dial-up connections to Verizon, MSN and all other ISPs using the UUNet backbone.")

UUNet has a service guarantee that it failed to meet: http://www.uu.net/customer/sla/ and it says that you are entitled to a credit. I believe you should pass this credit on to your customers who have been suffering from this problem.

While it appears to me now that Oceanic/Roadrunner are not the only ones that didn't recognize the problem, I think you need to re-evaluate your network monitoring procedures.

Aloha, Richard Gamberg