Trouble Beyond the Modem - Understanding Traces

Understanding Traces - II

The Microsoft-provided DOS trace tool is inadequate: it sends only 3 pings to each router, and does so sequentially after performing a reverse DNS on each router. When there is no reverse DNS entry for any particular router, the pings to all routers beyond are delayed by several seconds - and conditions anywhere along the route can change dramatically in several seconds. Packet loss is generally considered to be unacceptable if it exceeds 5%, but with only 3 samples, packet loss can only inaccurately be indicated as 0, 33, 66, or 100%. (An asterik - * - indicates packet loss in the trace below.)

Consider this DOS trace output:
Tracing route to eqworld-80.989studios.com [xxx.xxx.xxx.xxx]
over a maximum of 30 hops:
  1     7 ms     8 ms    10 ms  10.27.178.1
  2    10 ms    11 ms     8 ms  sc-24-24-192-78.socal.rr.com [24.24.192.78]
  3    20 ms    19 ms    22 ms  sc-24-24-192-81.socal.rr.com [24.24.192.81]
  4    26 ms     *       29 ms  sc-24-24-192-65.socal.rr.com [24.24.192.65]
  5    22 ms    24 ms    18 ms  sc-24-24-192-2.socal.rr.com [24.24.192.2]
  6    25 ms    19 ms    32 ms pos5-1-0.chswca1-rtr2.we.mediaone.net
[24.130.0.181]
  7    41 ms    27 ms    14 ms ser1-0.lsanca1-gsr2.we.mediaone.net
[24.130.0.205]
  8    34 ms     *        *     12.125.98.13
  9    26 ms    25 ms    29 ms  gbr2-p100.la2ca.ip.att.net [12.123.28.94]
 10    22 ms    37 ms    20 ms  gbr4-p70.la2ca.ip.att.net [12.122.1.201]
 11    30 ms    36 ms    43 ms  gbr2-p30.sd2ca.ip.att.net [12.122.2.121]
 12   201 ms   177 ms     *     gar1-p370.sd2ca.ip.att.net [12.123.145.25]
 13   171 ms     *        *     pos2-0.sdterman-2.sonyonline.net
[12.124.23.30]
 14   121 ms   149 ms   142 ms  eqworld-80.989studios.com [64.37.149.5]
Does the problem start at hop 4 - the first indication of packet loss or at hop 12 where the latency increases by 700%? Is the problem with the customer's provider (in this case, SoCal Road Runner), or in AT&T's network?

Here's a Ping Plotter screen with 3 samples:

No real problem is evident. Average latency Hawaii to California (hop 9) at 83ms, and then to Boston at 177ms. Could be better, but not bad, right?

Take a look at the same trace expanded to include 30 samples:

Look at hops 8 & 9....

Now, expanded to 300 samples, it's apparent that problems starting at hop 8 are causing trouble.

Finally, the results of 12 hours of monitoring with summary graph:

So where is the trouble, and who is responsible? If your answer is the trouble starts in UUNet's network at hop 8, or that UUNet is responsible, you need to read on.

NEXT PAGE - Understanding Traces III