I’m staying in the office very late tonight to write about a wrap-up of the past 3 days and the network issues we have been experiencing.  What we thought was a DDOS attack we found just wasn’t so.  We have just started to feel the crunch of consumer network providers trying to skimp out on their network costs and because of the saturation we saw on the network, our first suspicion was “DDOS”. After doing some extensive research and testing, the cause of the issue is a lot more startling.

Our main network provider is Level 3 Communications.  These guys are the best of the best and provide internet infrastructure (connectivity to ISP’s) to over 75% of the world.  Now, let that sink in a moment.  Before you go blasting this article for that statement, keep this in mind, our connectivity overseas is provided by Level 3.  Most commercial ISP in the USA is provided by Level 3.

About a month ago an article was written by a gentleman at BGR that reported the CEO of Level 3 chastised many consumer internet providers for, “purposely harming their customers”. Why? Because of their refusal to upgrade their peering hardware.

[note] Public Peering between two providers occurs when internet traffic from one provider is exchanged with another. Locations in the United States are limited to less than 100 cities that provide these facilities, and the high-bandwidth ports available for this peering is limited to less than 2,000 for all involved providers. [/note]

Currently most consumer grade providers are only offering 10Gbps ports at the public peering points in the United States, and are suffering from over 90% saturation which means home users of providers this CEO flagged experience much slower and unstable internet. I ran several tests over the course of 2 days after tagging this as a peering network issue and presented this data to both Level 3 and Time Warner Cable. Obviously, from everyone I have spoken with at Time Warner Cable believes it is necessary to reboot my modem, then send a technician out to replace it.

[toggle style=”toggle_icon” status=”closed” title=”March 27, 2015 Data”] [p] [code]

Tracing route to [74.50.105.2]
over a maximum of 30 hops:
0 xxxxxxxxxx [192.168.0.2]
1 192.168.0.1
2 cpe-071-077-000-001.nc.res.rr.com [71.77.0.1]
3 a-rtr2-10gig0-2-0-1.2123.nc.rr.com [66.26.46.121]
4 cpe-024-025-045-010.ec.res.rr.com [24.25.45.10]
5 be31.chrcnctr01r.southeast.rr.com [24.93.64.186]
6 bu-ether34.atlngamq46w-bcr00.tbone.rr.com [107.14.19.48]
7 0.ae5.pr0.atl20.tbone.rr.com [107.14.19.99]
8 atl-peer-01.twtelecom.net [198.32.182.17]
9 tpa1-ar1-xe-2-0-0-0.us.twtelecom.net [66.192.251.214]
10 66.162.27.46
11 10.253.0.238
12 [74.50.105.2]

Computing statistics for 300 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 xxxxxxxx [192.168.0.2]
0/ 100 = 0% |
1 6ms 0/ 100 = 0% 0/ 100 = 0% 192.168.0.1
0/ 100 = 0% |
2 29ms 0/ 100 = 0% 0/ 100 = 0% cpe-071-077-000-001.nc.res.rr.com [71.77.0.1]
0/ 100 = 0% |
3 32ms 0/ 100 = 0% 0/ 100 = 0% a-rtr2-10gig0-2-0-1.2123.nc.rr.com [66.26.46.121]
0/ 100 = 0% |
4 27ms 0/ 100 = 0% 0/ 100 = 0% cpe-024-025-045-010.ec.res.rr.com [24.25.45.10]
0/ 100 = 0% |
5 34ms 0/ 100 = 0% 0/ 100 = 0% be31.chrcnctr01r.southeast.rr.com 24.93.64.186]
0/ 100 = 0% |
6 37ms 0/ 100 = 0% 0/ 100 = 0% bu-ether34.atlngamq46w-bcr00.tbone.rr.com [107.14.19.48]
0/ 100 = 0% |
7 41ms 0/ 100 = 0% 0/ 100 = 0% 0.ae5.pr0.atl20.tbone.rr.com [107.14.19.99]
6/ 100 = 6% |
8 --- 100/ 100 =100% 94/ 100 = 94% atl-peer-01.twtelecom.net [198.32.182.17]
0/ 100 = 0% |
9 --- 100/ 100 =100% 94/ 100 = 94% tpa1-ar1-xe-2-0-0-0.us.twtelecom.net [66.192.251.214]
0/ 100 = 0% |
10 58ms 13/ 100 = 13% 7/ 100 = 7% 66.162.27.46
0/ 100 = 0% |
11 --- 100/ 100 =100% 94/ 100 = 94% 10.253.0.238
0/ 100 = 0% |
12 57ms 6/ 100 = 6% 0/ 100 = 0% [74.50.105.2]

Trace complete.

[/code] [/p] [/toggle]

[toggle style=”toggle_icon” status=”closed” title=”March 28, 2015 Data”] [p] [code]

Tracing route to [74.50.105.2]
over a maximum of 30 hops:
  0  xxxxxxxx [192.168.0.10]
  1  192.168.0.1
  2  cpe-071-077-000-001.nc.res.rr.com [71.77.0.1]
  3  a-rtr2-10gig0-2-0-1.2123.nc.rr.com [66.26.46.121]
  4  cpe-024-025-045-010.ec.res.rr.com [24.25.45.10]
  5  be31.chrcnctr01r.southeast.rr.com [24.93.64.186]
  6  bu-ether44.atlngamq46w-bcr00.tbone.rr.com [107.14.19.46]
  7  0.ae5.pr0.atl20.tbone.rr.com [107.14.19.99]
  8  atl-peer-01.twtelecom.net [198.32.182.17]
  9  tpa1-ar1-xe-2-0-0-0.us.twtelecom.net [66.192.251.214]
 10  66.162.27.46
 11  [74.50.105.2]

Computing statistics for 275 seconds...
            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address
  0                                           xxxxxxx[192.168.0.10]
                                0/ 100 =  0%   |
  1    5ms     0/ 100 =  0%     0/ 100 =  0%  192.168.0.1
                                0/ 100 =  0%   |
  2   23ms     0/ 100 =  0%     0/ 100 =  0%  cpe-071-077-000-001.nc.res.rr.com [71.77.0.1]
                                0/ 100 =  0%   |
  3   30ms     0/ 100 =  0%     0/ 100 =  0%  a-rtr2-10gig0-2-0-1.2123.nc.rr.com [66.26.46.121]
                                0/ 100 =  0%   |
  4   24ms     0/ 100 =  0%     0/ 100 =  0%  cpe-024-025-045-010.ec.res.rr.com [24.25.45.10]
                                0/ 100 =  0%   |
  5   33ms     0/ 100 =  0%     0/ 100 =  0%  be31.chrcnctr01r.southeast.rr.com[24.93.64.186]
                                0/ 100 =  0%   |
  6   35ms     0/ 100 =  0%     0/ 100 =  0%  bu-ether44.atlngamq46w-bcr00.tbone.rr.com [107.14.19.46]
                                0/ 100 =  0%   |
  7   38ms     0/ 100 =  0%     0/ 100 =  0%  0.ae5.pr0.atl20.tbone.rr.com [107.14.19.99]
                                4/ 100 =  4%   |
  8  ---     100/ 100 =100%    96/ 100 = 96%  atl-peer-01.twtelecom.net [198.32.182.17]
                                0/ 100 =  0%   |
  9  ---     100/ 100 =100%    96/ 100 = 96%  tpa1-ar1-xe-2-0-0-0.us.twtelecom.net [66.192.251.214]
                                0/ 100 =  0%   |
 10   55ms     4/ 100 =  4%     0/ 100 =  0%  66.162.27.46
                                1/ 100 =  1%   |
 11   54ms     5/ 100 =  5%     0/ 100 =  0%  [74.50.105.2]

Trace complete.

[/code] [/p] [/toggle]

These internet issues are unavoidable until these providers decide it is time to reinvest into better equipment at their peering points. This is an issue we should all be worried about, especially since the unveiling of the new Net Neutrality Regulations set by the FCC that demand that these providers stop throttling services which will add to even more internet saturation at these peers.