Frequent, brief outages on tserv4.nyc4 tunnel

[twinbee]

About once or twice a day, my tunnel twinbee-1.tunnel.tserv4.nyc4.ipv6.he.net loses connectivity for a minute or so.  During those times I am able to ping6 the server endpoint, but a traceroute reveals that the packets don't get any further than that.

I assume that if this were an issue with my connectivity or my configuration, I would be unable to reach the other end of the tunnel.  (Is that a reasonable assumption?)  I'd prefer to verify that the issue isn't on my end before I officially bug HE support about it.

Has anyone with tunnels on the same server experienced the same issue?  Thanks.

[kasperd]

I assume that if this were an issue with my connectivity or my configuration, I would be unable to reach the other end of the tunnel.  (Is that a reasonable assumption?)

From your description it does sound like the tunnel server has a connectivity problem with the rest of the HE network.

Has anyone with tunnels on the same server experienced the same issue?

I don't have any tunnels on that tunnel server. I'll try a periodic traceroute towards your IP to see if I can capture an output of what the situation looks like from the other end when the problem occurs.

This is what the traceroute from my end looks like right now, where the connection is working:

Code Select Expand

traceroute to twinbee-1-pt.tunnel.tserv4.nyc4.ipv6.he.net (2001:470:1f06:13b::2), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.668 ms  1.098 ms  2.401 ms
2  2001:470:27:940::1  35.284 ms  39.439 ms  45.505 ms
3  2001:470:0:11e::1  49.641 ms  40.723 ms  41.607 ms
4  2001:470:0:22f::1  56.143 ms  60.358 ms  65.449 ms
5  2001:470:0:3f::1  72.558 ms  69.322 ms  70.277 ms
6  2001:470:0:128::1  139.692 ms  142.498 ms  149.447 ms
7  2001:470:20::2  155.211 ms  141.553 ms  145.841 ms
8  2001:470:1f06:13b::2  153.180 ms  166.884 ms  155.562 ms

[twinbee]

Thanks, much obliged!

[cholzhauer]

I never see any problems, but it's my home tunnel and I'm not constantly on it.  How long does the outage last?

[kasperd]

Around 1:12 CEST I saw an outage lasting 1-2 minutes:

Code Select Expand

traceroute to twinbee-1-pt.tunnel.tserv4.nyc4.ipv6.he.net (2001:470:1f06:13b::2), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.560 ms  26.473 ms  26.992 ms
2  2001:470:27:940::1  130.609 ms  111.741 ms  111.732 ms
3  2001:470:0:11e::1  117.794 ms  27.059 ms  30.887 ms
4  2001:470:0:22f::1  75.144 ms  76.812 ms  76.789 ms
5  2001:470:0:3f::1  71.083 ms  71.530 ms  71.941 ms
6  2001:470:0:128::1  127.003 ms  132.490 ms  129.562 ms
7  *  *  *
8  *  *  *
9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *That was the only outage lasting more than 30 seconds. At 6:22 I saw this:

Code Select Expand

traceroute to twinbee1-pt.tunnel.tserv4.nyc4.ipv6.he.net (2001:470:1f06:13b::2), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.640 ms  26.630 ms  29.656 ms
2  2001:470:27:940::1  88.500 ms  68.132 ms  68.930 ms
3  2001:470:0:11e::1  71.353 ms  31.957 ms  35.318 ms
4  2001:470:0:22f::1  59.966 ms  65.083 ms  64.553 ms
5  2001:470:0:3f::1  81.507 ms  78.830 ms  79.176 ms
6  2001:470:0:128::1  125.480 ms  137.253 ms  134.440 ms
7  *  2001:470:20::2  3013.035 ms  *
8  *  *  *
9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  2001:470:1f06:13b::2  2979.384 ms  2859.134 msAfter that I saw multiple short outages, which appeared to be on the IPv4 side. The first one was at 7:19

Code Select Expand

traceroute to twinbee-1-pt.tunnel.tserv4.nyc4.ipv6.he.net (2001:470:1f06:13b::2), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.549 ms  21.125 ms  21.693 ms
2  2001:470:27:940::1  36.378 ms  24.820 ms  29.337 ms
3  2001:470:0:11e::1  32.421 ms  26.103 ms  39.914 ms
4  2001:470:0:22f::1  57.258 ms  58.040 ms  53.628 ms
5  2001:470:0:3f::1  63.488 ms  64.182 ms  68.110 ms
6  2001:470:0:128::1  132.173 ms  145.844 ms  132.615 ms
7  2001:470:20::2  133.246 ms  124.760 ms  128.237 ms
8  *  *  *
9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *There clearly were two outages on the IPv6 side. And combined with your observations, I think it would be appropriate to email ipv6@he.net about it. There also appears to have been a few outages on the IPv4 side. With the information I have access to I cannot say if those outages are on the HE side, on your side, or somewhere in between. It is not impossible that there is one root cause responsible for both sorts of outages.

[kasperd]

I just noticed that I still have the command running to periodically ping your tunnel. The brief outages still appear to be happening. If it is any use to you, I can attach the full output to this thread or email it to you, it is 90MB (compressed to 5MB with gzip).

[kcochran]

Metrics involving the tservs at either nyc4 or tor1 right now are going to be unusual, as tor1 is acting as the nyc4 tserv right now, and taking more tunnels than it ever normally should.

nyc4 (and by extension tor1) should be back to normal in another week or so, time specifics variable due to Sandy-effects.

[kasperd]

tor1 is acting as the nyc4 tserv right now

The issue had been ongoing before that. But it may not make sense to investigate that issue before nyc4 is back. I'll just leave the command running for now.

[kcochran]

tserv4.nyc4 is back in normal operation on new hardware.

[kmp6]

tserv4.nyc4 is back in normal operation on new hardware.

Guys,

I'd been wondering about the latency increase to this tunnel... and it's now back to where it was.  Thanks very much!  Life is good again...

_KMP