This is Modemsite.com
Modemsite.com - Helping you get the best performance from your modem
 Troubleshooting News Technical Search
 Home Forum 56 Premium Site Map
Home Broadband - Not So Fast   Road Runner Nightmare

Roadrunner Nightmare

The problem discussed below was finally corrected late on December 30 (although the Roadrunner operation is still clueless). Turns out that the problem in UUNet has been affecting all Hawaii ISPs that use UUNet for connectivity since at least November! Details of the resolution of the nightmare are on the last page.

December 27, 2000: For more than a week, my connection to the Internet has been driving me crazy - most sites come in must fine, but there are some that are effectively useless. December 28 - no change. December 29 - now even worse - takes 4 minutes 12 second to load the Zonelabs page! December 30 - no change.

This problem affects all Roadrunner West-Hawaii customers - but the affected sites differ from customer to customer. (This is because one of the techniques used in Internet backbone routing uses the IP addresses at each end of the link in a hashing algorithm to select one of multiple paths to route traffic through.)  And unlike most Internet slowdowns, this one is constant - 24hours a day 7 days a week unless you get a different IP address. (With dial-up service, a user is likely to get a different IP address on each connection; with a broadband connection, your IP address rarely changes.)

The problem is best described with an analogy: Imagine if whenever you used your telephone to dial your Mother in LA, you get a busy signal. Maybe once in 20 tries, the call goes through, but you get disconnected. Go to your neighbor's house, and you have no trouble! But, your neighbor can't call their family in LA, but can from your phone!7

This all started around December 15 when Roadrunner Hawaii added another DS-3 (45mbps) link to the Internet from a Tier-1 provider, and started "balancing" the network. The preferred routing for servers sending data to West Hawaii users was changed from AT&T to UUNet. [While Roadrunner uses many Tier-1 providers, and outbound data may travel via any of these, the data that comes back to you will route independently and is based upon your public IP address. In Hawaii, Roadrunner assigns a large block of users to addresses that have a common preferred inbound route of either AT&T or UUNet, and "balancing" has involved changing which carriers serve different users: think rolling blackouts that last for weeks.] There are 2 out of at least 12 UUNet routers that introduce extreme latency and packet loss 24 hours a day 7 days a week. The rest work fine. [Unbalanced network.] Unfortunately, the path taken from any particular site to your particular IP will rarely change - so if the route is through a bad router, you'll get less than 28k modem speeds!

For example, right now (27-Dec) it takes 2 minutes and 44 seconds to load www.zonelabs.com !

Look at the trace to the site:

First, ignore the 100% loss and unidentified router - it is normal to encounter routers that do not respond to ICMP packets. Note the latency at all hops except 152.63.53.93 and the Zonelabs server are excellent.

The problem is really not identified well because the routers used in the return path back to me are not identified in a trace. Routers along the way can take different paths on the way back, and I think this is what is happening here. The problem is more easily seen doing a reverse trace. There is a directory of sites with reverse trace tools at www.traceroute.org .

From : www.netnostics.com/tracert.cgi

To my IP:
Result for a24b165n36client148.hawaii.rr.com: 
traceroute to a24b165n36client148.hawaii.rr.com (24.165.36.148), 30 hops max, 40 byte packets
 1  netnostics (63.64.190.1)  1.107 ms  0.960 ms  0.973 ms
 2  Serial1-1-1.GW3.DCA1.ALTER.NET (157.130.16.57)  0.896 ms  3.618 ms  1.640 ms
 3  522.ATM1-0.XR2.DCA1.ALTER.NET (152.63.37.66)  1.703 ms  1.122 ms  1.024 ms
 4  194.at-0-0-0.TR2.DCA8.ALTER.NET (152.63.32.174)  1.546 ms  1.427 ms  1.844 ms
 5  115.at-6-1-0.TR4.SCL1.ALTER.NET (152.63.3.246)  62.654 ms  62.692 ms  62.536 ms
 6  399.ATM6-0.XR2.PAO1.ALTER.NET (152.63.49.25)  66.238 ms  65.140 ms  65.208 ms
 7  188.ATM10-0-0.CR1.PAO1.ALTER.NET (146.188.147.189)  2162.787 ms  2688.173 ms  2532.479 ms
 8  197.Hssi4-0.GW1.HAW2.ALTER.NET (137.39.74.117)  2999.107 ms  2832.314 ms  2715.960 ms
 9  * * *
10  24.25.225.81 (24.25.225.81)  2462.390 ms  2378.096 ms  2597.500 ms
11  24.25.225.157 (24.25.225.157)  2503.834 ms  2464.486 ms  2326.038 ms
12  24.25.225.170 (24.25.225.170)  2118.566 ms  2131.334 ms  2159.567 ms
13  24.25.225.169 (24.25.225.169)  1977.412 ms  2143.371 ms  2075.853 ms
14  24.25.225.134 (24.25.225.134)  2003.156 ms  2252.479 ms  2206.641 ms 

In the above reverse trace, notice hop 7 - PAO= Palo Alto, California; Alter.net = UUNet/MCI Worldcom. This router shows over 2 seconds of latency, and affects all routers beyond it back to me.

Now, a traceroute from the same Netnostics server to my an IP only 1 digit away from mine - .147 at the end instead of .148:

Result for 24.165.36.147:
traceroute to 24.165.36.147 (24.165.36.147), 30 hops max, 40 byte packets
 1  netnostics (63.64.190.1)  0.902 ms  0.532 ms  0.500 ms
 2  Serial1-1-1.GW3.DCA1.ALTER.NET (157.130.16.57)  0.739 ms  0.762 ms  0.784 ms
 3  522.ATM3-0.XR2.DCA1.ALTER.NET (152.63.37.74)  1.090 ms  1.446 ms  1.854 ms
 4  194.at-0-0-0.TR2.DCA8.ALTER.NET (152.63.32.174)  1.563 ms  1.505 ms  1.285 ms
 5  115.at-6-1-0.TR4.SCL1.ALTER.NET (152.63.3.246)  62.572 ms  63.878 ms  62.547 ms
 6  399.ATM6-0.XR2.PAO1.ALTER.NET (152.63.49.25)  65.206 ms  65.720 ms  65.116 ms
 7  188.ATM10-0-0.CR2.PAO1.ALTER.NET (146.188.147.205)  65.098 ms  63.990 ms  66.468 ms
 8  197.Hssi5-0.GW1.HAW2.ALTER.NET (137.39.74.121)  114.217 ms  114.917 ms  114.970 ms
 9  * * *
10  24.25.225.81 (24.25.225.81)  117.399 ms  120.990 ms  117.616 ms
11  24.25.225.157 (24.25.225.157)  114.320 ms  114.521 ms  114.032 ms
12  24.25.225.170 (24.25.225.170)  123.707 ms  127.555 ms  141.346 ms
13  24.25.225.169 (24.25.225.169)  128.170 ms  125.774 ms  122.968 ms
14  24.25.225.134 (24.25.225.134)  125.154 ms  137.065 ms  127.237 ms
15  * *

There is no problem going to this IP! Notice hop 7 is a different IP address.

OK, so I think I've pinpointed the problem - why do I have to do this? And, even though I've located the problem, how can I do anything to get it fixed? (I also identified a second UUNet router at 146.188.147.177 with the same problem.)

Roadrunner/Oceanic Cable-Hawaii [Time Warner] is my ISP. I've got a ticket number. Supposedly "escalated". I have sent them traces and messages, and they haven't sent a single reply. 'All agents are busy - leave a message' on their phone line. I leave messages, but they don't call back.

Roadrunner 'National' Support online chat - Three sessions, and the best I can get out of them:
"...the router which is in trouble belongs to UUNET, you need to contact them at 800-900-0241 about this problem."

I call that number. There's no option that fits my situation, but I try their tech support anyway. The agent tells me "I'm lucky", he can see a problem and asks me to e-mail traces to help@uu.net which I do. They even create a ticket number, and send an automated response to me. 18 hours later, I call back and ask about the ticket number..... I'm told that I'm not a UUNet customer, and they can't help me - Roadrunner needs to contact them. Good luck!

Now, I try the technical contact for zonelabs.com. I get a prompt reply - they don't see a problem! Well, of course they don't because they're tracing from a different IP than the zonelabs server. They operate Brainstorm.com, and I can access that just fine:

While the UUNet router in the middle of the trace still shows trouble (because it is using a 'bad' return path to me), it is not slowing down data moving to Brainstorm; the Brainstorm server's data to me gets routed over a good (different) path back to me! Zonelabs' provider - RCN (who purchased Erol's) actually got back to me, and looked further into the problem. Yes, they see the problem, but although they use UUNet as their provider, the problem is not between them and UUNet, it's between Roadrunner and UUNet - Roadrunner must address the problem.

Since Oceanic Cable/Roadrunner Hawaii still will not even acknowledge the problem, I call Roadrunner's NOC in Herndon, Virginia. On December 26, I'm told they are aware of a problem - that it affects many Roadrunner systems, not just Hawaii, and that their engineers are meeting and will fix the problem. December 29 - still not fixed and I call back again. After 30 minutes on the phone, I convince the tech there's a problem, and he promises to look into it even though he 'could get in trouble' for talking to me - they are only supposed to deal with local Roadrunner Help Desks.

It boggles my mind that first of all UUNet and Roadrunner cannot acknowledge a problem - the first step in fixing it! It boggles my mind that I'm stuck in the middle and this situation - which already has been ongoing for more than a week - has no end in sight!

If I reboot everything including my cable modem, and if I am assigned a different IP address, Zonelabs.com will come in just fine - but then I'm able to start building a new list of sites that act like Zonelabs, because a new set of sites will choose the bad UUNet path back to me!

For example, when I was assigned IP 24.165.37.98, www.dslreports.com/tweaks was affected:

Ping stability 2289 152 - - - - - - - 154
* Wide variation in ping, 154 to 2289
* Unusually high average ping time

Out of 10 pings, only 3 made it. But, with the IP address I currently lease, no problem:

Ping stability 158 145 153 180 147 155 148 152 164 148
* Quick packet-loss tested ok

Click here for MORE traces

Home  |  Links  |  Send Feedback  |  Privacy Policy  | Report Broken Link
Legal Page  |  Author's Web Sites   |  Log In   
 

Modemsite.com 1998-2016 v.Richard Gamberg. All rights reserved.