cogentco randomness with hurricane electric

[gnarlymarley]

I am wonder if anyone has noticed this, but the connection between Hurricane Electric and cogentco is up and down over the period of over a day.  What I have noticed is sometimes, however rare, we are able to reach cogentco over the he.net tunnels.  I also noticed that 100% of the time the freenet6 tunnels always reach cogentco.  I have included a trace below for anyone interesting in helping debug this issue from 5:22am Central Time.  Attempts to contact either party have failed due to their systems working successfully over IPv6.  My question is, does anyone know where or how we can fix this brokenness?  As near as I can tell, the issue lies somewhere around ntt.net, but I am not able to prove it.  Many of the looking-glass servers around are able to reach cogentco successfully.  Is it possible that they are turning BGP on and off, or is it more likely that there is a bad network cable causing the issues?

traceroute6 to 2001:550:1:b::d (2001:550:1:b::d) from 2001:470:1f04:170b::2, 64 hops max, 16 byte packets
1  2001:470:1f04:170b::1  52.337 ms  46.367 ms  39.631 ms
2  2001:470:0:45::1  44.550 ms  49.645 ms  49.595 ms
3  2001:470:0:31::2  44.555 ms  49.607 ms  47.149 ms
4  2001:470:0:1b4::2  74.562 ms  67.068 ms  64.682 ms
5  2001:470:0:1af::1  87.115 ms  89.525 ms  99.701 ms
6  2001:470:0:286::2  109.598 ms  104.823 ms  104.379 ms
7  2001:470:0:1b1::2  184.626 ms  182.095 ms  187.150 ms
8  2001:7f8:43::2:1371:1  182.064 ms  184.584 ms  184.373 ms
9  2001:728:0:5000::149  189.418 ms  197.194 ms  212.125 ms
10  2001:728:0:2000::199  192.282 ms  194.431 ms  209.572 ms
11  2001:728:0:2000::1a6  194.548 ms  194.782 ms  191.979 ms
12  2001:978:3::41  179.561 ms  182.334 ms  184.430 ms
13  2001:978::19  249.637 ms  254.594 ms  252.223 ms
14  2001:550::105  299.338 ms  249.705 ms  257.103 ms
15  2001:550::26  247.079 ms  252.123 ms  249.828 ms
16  2001:550::76  256.927 ms  249.578 ms  249.578 ms
17  2001:550::212  249.603 ms  254.610 ms  254.538 ms
18  2001:550::141  255.505 ms  249.586 ms  247.136 ms
19  2001:550:1:b::1  252.106 ms  252.075 ms  249.738 ms
20  2001:550:1:b::d  259.470 ms  257.086 ms  257.130 ms

Tracing route to auth2.dns.cogentco.com [2001:550:1:b::d]
over a maximum of 30 hops:

  1   100 ms    96 ms    93 ms  2001:5c0:1000:b::3b1e
  2   149 ms   109 ms   102 ms  ix-5-0-1.6bb1.MTT-Montreal.ipv6.as6453.net [2001:5a0:300::5]
  3    94 ms    96 ms    90 ms  if-ge-11-3-0.0.tcore2.MTT-Montreal.ipv6.as6453.net [2001:5a0:1900:100::d]
  4    98 ms   146 ms   114 ms  if-ae2.2.tcore2.NYY-NewYork.ipv6.as6453.net [2001:5a0:1900:100::6]
  5   135 ms   104 ms   112 ms  if-ae11.2.tcore1.NYY-NewYork.ipv6.as6453.net [2001:5a0:400:700::2]
  6   101 ms   107 ms   109 ms  if-ae5.5.tcore1.NTO-NewYork.ipv6.as6453.net [2001:5a0:400:200::e]
  7   106 ms    99 ms   102 ms  if-ae8.2.tcore2.NTO-NewYork.ipv6.as6453.net [2001:5a0:12:100::2e]
  8   124 ms   116 ms   112 ms  2001:550:3::f9
  9   188 ms   190 ms   182 ms  2001:550::105
10   232 ms   206 ms   195 ms  2001:550::26
11   190 ms   191 ms   194 ms  2001:550::76
12   187 ms   187 ms   191 ms  2001:550::212
13   200 ms   199 ms   191 ms  2001:550::141
14   190 ms   191 ms   179 ms  2001:550:1:b::1
15   190 ms   179 ms   181 ms  auth2.dns.cogentco.com [2001:550:1:b::d]

Trace complete.

[broquea]

Sounds like someone either leaking a route to HE, or someone announcing a specific:

Code Select Expand

Connected to route-server.he.net.
route-server> sh ipv6 bgp 2001:550::/32
% Network not in table


Code Select Expand

HOST: ipvsixme                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- f0-6.switch14.fmt2.he.net  0.0%     5    0.6   0.6   0.5   0.6   0.0
  2.|-- 10ge3-3.core3.fmt2.he.net  0.0%     5    0.5   0.5   0.5   0.6   0.0
  3.|-- 10ge5-4.core1.pao1.he.net  0.0%     5   10.5   5.3   1.2  12.5   5.7
  4.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0


[plugwash]

There has been a disconnect between HE and cogent on ipv6 for ages. AIUI HE see themselves as an "IPv6 tier 1" and as such refuse to buy transit to reach cogent and cogent will not peer with HE. I do not belive cogent have pubilcally stated why they won't peer with HE but I suspect they see HE as an upstart with ideas above their station.

In your first trace the packets seem to go across equninix paris (2001:7f8:43::/48) to NTT (2001:728::/48) to cogent (2001:978::/32).

This looks like a strange route, if HE were buying transit from NTT to reach cogent then I wouldn't expect them to do it via an IX and cogent would have no reason to buy transit to reach HE when they could just peer with them.

I think the most likely explanation is that someone at NTT screwed up and inadvertantly established a route between HE and cogent that was never supposed to exist.