tunnel tserv3.fmt2.ipv6.he.net down?

[cam]

Hello,

I have 2 tunnels:
Tunnel [ 2 / 5 ]   Routed /64   Routed /48   Description
XXX.tunnel.tserv3.fmt2.ipv6.he.net    2001:470:XXX:79d::/64 2001:470:XXX::/48   test1
YYY.tunnel.tserv25.sin1.ipv6.he.net 2001:470:YYY:5b9::/64 2001:470:YYY::/48 test2

Configuring the remote / local endpoint on Openbsd 5.1 for test1 results in the impossibility to ping the Server IPv6 Address tunnel endpoint (gateway) nor the www.kame.net site (no route to host even thought the route is here)
Doing the exact same configuration (updated the local IPv4 to the correct one etc.) with the tunnel test2 gives no issues at all. I can ping the ipv6 tunnel endpoint=gateway and ping6 www.kame.net
changing the configuration back to test1 does not work at all.

Since the only settings changing in the conf are the tunnel endpoints (ipv6 and ipv4), i suppose the POP tunnel tserv3.fmt2.ipv6.he.net down? has a problem??

What steps could be done?

cheers,
cam

[cholzhauer]

If you think it's down, email ipv6@he.net

[kriteknetworks]

http://tunnelbroker.net/status.php

[cam]

hello,

yes i have seen that page .. but when changing to another POP, I get my tunnel working.. so I am wondering where is the issue..

anyway i sent an email to ipv6@he.net. will keep you posted if it gets solved.

[kasperd]

This is what I see when I run traceroute6 to the first tunnel:

Code Select Expand

1  2001:470:1f0b:1e45:1f9c:c1d2:c8b6:9008  0.187 ms  0.178 ms  0.218 ms
2  2001:470:1f0a:1e45::1  66.271 ms  72.681 ms  78.125 ms
3  2001:470:0:69::1  79.422 ms  51.159 ms  52.721 ms
4  2001:470:0:1d2::1  75.283 ms  76.369 ms  70.965 ms
5  2001:470:0:128::1  140.377 ms  139.765 ms  139.787 ms
6  2001:470:0:10e::1  199.097 ms  199.133 ms  212.309 ms
7  2001:470:0:18d::1  212.338 ms  224.666 ms  224.688 ms
8  2001:470:0:23b::2  223.450 ms  223.466 ms  223.449 ms
9  2001:470:0:45::2  223.571 ms  205.924 ms  209.767 ms
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *

And this is what I see when I run traceroute6 to the other tunnel:

Code Select Expand

1  2001:470:1f0b:1e45:1f9c:c1d2:c8b6:9008  0.207 ms  0.192 ms  0.233 ms
2  2001:470:1f0a:1e45::1  60.889 ms  67.322 ms  71.500 ms
3  2001:470:0:69::1  72.342 ms  44.881 ms  45.785 ms
4  2001:470:0:1d2::1  83.327 ms  83.648 ms  83.579 ms
5  2001:470:0:128::1  163.153 ms  162.404 ms  162.442 ms
6  2001:470:0:10e::1  229.488 ms  229.550 ms  228.696 ms
7  2001:470:0:269::2  381.164 ms  342.547 ms  343.337 ms
8  2001:470:0:26b::2  385.815 ms  385.886 ms  382.739 ms
9  2001:470:20::2  381.807 ms  395.487 ms  395.538 ms
10  *  *  *
11  *  *  *
12  *  *  *
13  *  2001:470:20::2  376.518 ms !H  381.796 ms !H

The result looks a little bit different, but I don't get a reply from your gateway on any of the two. If I do a traceroute to the server end of the tunnel, I do get a reply in both cases.

There are so many possible explanations, and since I have no access to either end of the tunnel, I cannot do much more to debug this. I suggest you try to run a traceroute to the IPv4 address of each tunnel server and post the results, so we can say if that provides any hints.

[cam]

Hello,

thank you for your reply.
I sent that email to ipv6@he.net and then went to sleep.
This morning when I woke up, I got an email from ipv6@he.net (very quick !! really great) saying they dont see any issues on their end nor did any changes.
I did some ping / ping6 and well, everything is back to normal and I didnt change anything as well..
To be honest, not sure what happened in between...

Cheers,
cam

[kasperd]

To be honest, not sure what happened in between...

My guess is some problem on the IPv4 connection between the two endpoints of the tunnel. An IPv4 traceroute at the time where it was going on may have revealed where on the path it happened.

We'll never know for sure unless it happens again.

[cam]

I havent done a traceroute between my ipv4 and the remote ipv4.
However I did manage to ping remote_ipv4 without any problem.

I will keep that in mind for next time..

thanks

[kasperd]

However I did manage to ping remote_ipv4 without any problem.

Then a standard traceroute will probably not tell you much more, except if the problem happens to be correlated with a change in the route. There is another possibility, but it will require much more specialized tools.

A traceroute with proper 6in4 packets could work the following way. The IPv6 packet is constructed as an ordinary and fully valid packet leaving your network, but with a hop limit of just one. This IPv6 packet is then inserted into IPv4 packets with increasing TTL.

Then you will get ordinary time to live exceeded errors until you reach the tunnel server, at which point you should immediately get an ICMPv6 error indicating the hop limit has expired on the IPv6 packet. While performing such a traceroute keep an eye out for all the ICMP and ICMPv6 errors that you get back.

A lot of work, but may reveal if something really strange is going on. An example of what strange things you can find with a method similar to the above is a router that instead of sending a time to live expired message will take the encapsulated IPv6 packet and forward that instead.

[cam]

Hello,

Opening this thread again..
Since my first post, I faced it 4 more times and right now again. My ipv4 local endpoint is set to my correct external IPv4 address.
Any idea to diagnose the issue?

A bit of troubleshooting below:

Traceroute to my ipv6 local endpoint

Code Select Expand

# traceroute6 2001:470:1f04:79d::2
traceroute6 to 2001:470:1f04:79d::2 (2001:470:1f04:79d::2) from 2001:470:1f04:79d::2, 64 hops max, 12 byte packets
1  cam-2-pt.tunnel.tserv3.fmt2.ipv6.he.net  1.833 ms  0.605 ms  0.543 ms


Traceroute to my ipv6 remote endpoint

Code Select Expand

# traceroute6 2001:470:1f04:79d::1
traceroute6 to 2001:470:1f04:79d::1 (2001:470:1f04:79d::1) from 2001:470:1f04:79d::2, 64 hops max, 12 byte packets
1  * * *
2  * * *
3  *^C


my ifconfig setup

Code Select Expand

# ifconfig gif0
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
        priority: 0
        groups: gif egress
        physical address inet 192.168.1.253 --> 72.52.104.74
        inet6 fe80::200:24ff:fec9:a188%gif0 ->  prefixlen 64 scopeid 0xc
        inet6 2001:470:1f04:79d::2 -> 2001:470:1f04:79d::1 prefixlen 128


ping to my remote ipv4 endpoint

Code Select Expand

# ping 72.52.104.74
PING 72.52.104.74 (72.52.104.74): 56 data bytes
64 bytes from 72.52.104.74: icmp_seq=0 ttl=49 time=207.537 ms
64 bytes from 72.52.104.74: icmp_seq=1 ttl=49 time=210.686 ms
64 bytes from 72.52.104.74: icmp_seq=2 ttl=49 time=210.971 ms
64 bytes from 72.52.104.74: icmp_seq=3 ttl=49 time=196.691 ms
64 bytes from 72.52.104.74: icmp_seq=4 ttl=51 time=193.147 ms
--- 72.52.104.74 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 193.147/203.806/210.971/7.451 ms


traceroute to remote ipv4 endpoint

Code Select Expand

# traceroute -n 72.52.104.74
traceroute to 72.52.104.74 (72.52.104.74), 64 hops max, 40 byte packets
1  192.168.1.254  1.953 ms  1.903 ms  1.839 ms
2  116.14.192.1  27.174 ms  21.562 ms  24.817 ms
3  202.166.120.121  21.38 ms  20.890 ms  20.625 ms
4  202.166.126.221  50.590 ms  21.110 ms  21.173 ms
5  202.166.119.234  21.87 ms  69.603 ms  21.123 ms
6  202.166.120.182  22.11 ms  22.48 ms  21.563 ms
7  202.166.126.97  22.133 ms  21.542 ms  22.138 ms
8  203.208.190.141  22.709 ms  21.781 ms  22.451 ms
9  203.208.166.57  22.85 ms 203.208.152.217  22.720 ms 203.208.166.197  22 ms
10  203.208.178.206  228.735 ms 203.208.153.126  206.52 ms 203.208.171.186  191.414 ms
11  203.208.171.242  217.662 ms 203.208.171.158  210.625 ms 203.208.152.110  217.656 ms
12  198.32.176.20  224.722 ms  207.886 ms  221.527 ms
13  72.52.92.66  231.611 ms  213.318 ms  201.556 ms
14  72.52.104.74  207.675 ms  206.94 ms  214.719 ms


tcpdump while doing a traceroute6 to www.kame.net

Code Select Expand

# tcpdump -vvv -n -e -ttt -i gif0 host 2001:470:1f04:79d::2
tcpdump: listening on gif0, link-type NULL
Sep 12 00:35:24.491455 2001:470:1f04:79d::2.42457 > 2001:200:dff:fff1:216:3eff:feb1:44d7.33435: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 00:35:29.500808 2001:470:1f04:79d::2.42457 > 2001:200:dff:fff1:216:3eff:feb1:44d7.33436: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 00:35:34.510478 2001:470:1f04:79d::2.42457 > 2001:200:dff:fff1:216:3eff:feb1:44d7.33437: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 00:35:39.521980 2001:470:1f04:79d::2.42457 > 2001:200:dff:fff1:216:3eff:feb1:44d7.33438: [udp sum ok] udp 12 (len 20, hlim 2)
Sep 12 00:35:44.530167 2001:470:1f04:79d::2.42457 > 2001:200:dff:fff1:216:3eff:feb1:44d7.33439: [udp sum ok] udp 12 (len 20, hlim 2)
Sep 12 00:35:49.540051 2001:470:1f04:79d::2.42457 > 2001:200:dff:fff1:216:3eff:feb1:44d7.33440: [udp sum ok] udp 12 (len 20, hlim 2)
^C
34 packets received by filter


tcpdump while performing a traceroute6 to my ipv6 remote endpoint

Code Select Expand

# tcpdump -vvv -n -e -ttt -i gif0 host 2001:470:1f04:79d::2
tcpdump: listening on gif0, link-type NULL
Sep 12 00:36:58.707170 2001:470:1f04:79d::2.19393 > 2001:470:1f04:79d::1.33435: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 00:37:06.357077 2001:470:1f04:79d::2.19393 > 2001:470:1f04:79d::1.33436: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 00:37:11.366920 2001:470:1f04:79d::2.19393 > 2001:470:1f04:79d::1.33437: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 00:37:16.378393 2001:470:1f04:79d::2.19393 > 2001:470:1f04:79d::1.33438: [udp sum ok] udp 12 (len 20, hlim 2)
Sep 12 00:37:21.386688 2001:470:1f04:79d::2.19393 > 2001:470:1f04:79d::1.33439: [udp sum ok] udp 12 (len 20, hlim 2)
Sep 12 00:37:26.396399 2001:470:1f04:79d::2.19393 > 2001:470:1f04:79d::1.33440: [udp sum ok] udp 12 (len 20, hlim 2)
Sep 12 00:37:31.407859 2001:470:1f04:79d::2.19393 > 2001:470:1f04:79d::1.33441: [udp sum ok] udp 12 (len 20, hlim 3)


traceroute6 from another server to ipv6 remote endpoint

Code Select Expand

# traceroute6 -n 2001:470:1f04:79d::1
traceroute6 to 2001:470:1f04:79d::1 (2001:470:1f04:79d::1) from 2XXX:XXXX:YYYY::X, 64 hops max, 12 byte packets
1  2XXX:XXXX:YYYY::Y  0.863 ms  1.271 ms  0.565 ms
2  2001:504:13::1a  1.073 ms  0.738 ms  0.794 ms
3  2001:470:0:72::1  0.917 ms  1.539 ms  9.245 ms
4  2001:470:0:18d::1  18.923 ms  8.816 ms  15.317 ms
5  2001:470:0:23b::2  58.726 ms  9.539 ms  14.186 ms
6  2001:470:1f04:79d::1  9.668 ms  9.666 ms  18.077 ms


traceroute6 from another server to my ipv6 local endpoint (tcpdump on gif0 does not capture anything from that other server)

Code Select Expand

# traceroute6 -n 2001:470:1f04:79d::2
traceroute6 to 2001:470:1f04:79d::2 (2001:470:1f04:79d::2) from 2XXX:XXXX:YYYY::X, 64 hops max, 12 byte packets
1  2XXX:XXXX:YYYY::Y  0.774 ms  0.62 ms  0.626 ms
2  2001:504:13::1a  0.845 ms  0.798 ms  0.871 ms
3  2001:470:0:72::1  0.807 ms  5.356 ms  0.793 ms
4  2001:470:0:18d::1  19.053 ms  8.867 ms  15.963 ms
5  2001:470:0:23b::2  9.367 ms  9.516 ms  14.527 ms
6  2001:470:20::2  17.173 ms  14.104 ms  14.277 ms
7  * * *
8  * * *
9  * * *
..snip..


[kasperd]

traceroute6 from another server to my ipv6 local endpoint (tcpdump on gif0 does not capture anything from that other server)

You should compare the tcpdump on gif0 (which based on your output must be the virtual interface) with a tcpdump on the physical interface, the tunnelled packets are send over. If the physical interface was called eth0, you could run tcpdump -pni eth0 'proto 41'

Based on your output, it does sound like a problem on the tunnel server. If tcpdump on the physical interface shows the same packets as tcpdump on the virtual interface, then I think the problem is on the tunnel server. If the problem persists, you should contact ipv6@he.net again.

[cam]

thanks for the reply.
Issue still persists even after overnight. I have tcpdump both on gif0 (virtual interface, you are right) and sis2 (my physical interface) while doing a traceroute6 -n 2001:470:1f04:79d::1
Performing a traceroute from my other server to my local ipv6 endpoint and performing the same tcpdump on both gif0 and sis2 does not capture any single packets.

I have emailed a few times the ipv6@he.net but they came back with the same exact answer "nothing is wrong on their side"
I am wondering what happens on my side then...


while doing a traceroute6 -n 2001:470:1f04:79d::1

Code Select Expand

# tcpdump -vvv -n -e -ttt -i gif0
tcpdump: listening on gif0, link-type NULL
Sep 12 09:07:02.575567 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33435: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 09:07:07.578553 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33436: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 09:07:12.588373 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33437: [udp sum ok] udp 12 [hlim 1] (len 20)
Sep 12 09:07:17.599771 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33438: [udp sum ok] udp 12 (len 20, hlim 2)
Sep 12 09:07:22.608064 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33439: [udp sum ok] udp 12 (len 20, hlim 2)
Sep 12 09:07:27.618793 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33440: [udp sum ok] udp 12 (len 20, hlim 2)
Sep 12 09:07:38.119455 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33441: [udp sum ok] udp 12 (len 20, hlim 3)
Sep 12 09:07:43.127722 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33442: [udp sum ok] udp 12 (len 20, hlim 3)
Sep 12 09:07:48.137565 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33443: [udp sum ok] udp 12 (len 20, hlim 3)
Sep 12 09:07:53.149109 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33444: [udp sum ok] udp 12 (len 20, hlim 4)


while doing a traceroute6 -n 2001:470:1f04:79d::1

Code Select Expand


# tcpdump -vvv -n -ttt -i sis2 'proto 41'
tcpdump: listening on sis2, link-type EN10MB
Sep 12 09:07:02.575764 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33435: [udp sum ok] udp 12 [hlim 1] (len 20) [tos 0x10] (ttl 64, id 2099, len 80)
Sep 12 09:07:07.578743 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33436: [udp sum ok] udp 12 [hlim 1] (len 20) [tos 0x10] (ttl 64, id 39099, len 80)
Sep 12 09:07:12.588564 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33437: [udp sum ok] udp 12 [hlim 1] (len 20) [tos 0x10] (ttl 64, id 30497, len 80)
Sep 12 09:07:17.599963 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33438: [udp sum ok] udp 12 (len 20, hlim 2) [tos 0x10] (ttl 64, id 16900, len 80)
Sep 12 09:07:22.608255 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33439: [udp sum ok] udp 12 (len 20, hlim 2) [tos 0x10] (ttl 64, id 38097, len 80)
Sep 12 09:07:27.618987 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33440: [udp sum ok] udp 12 (len 20, hlim 2) [tos 0x10] (ttl 64, id 64837, len 80)
Sep 12 09:07:38.119650 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33441: [udp sum ok] udp 12 (len 20, hlim 3) [tos 0x10] (ttl 64, id 55056, len 80)
Sep 12 09:07:43.127921 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33442: [udp sum ok] udp 12 (len 20, hlim 3) [tos 0x10] (ttl 64, id 33923, len 80)
Sep 12 09:07:48.137753 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33443: [udp sum ok] udp 12 (len 20, hlim 3) [tos 0x10] (ttl 64, id 63147, len 80)
Sep 12 09:07:53.149300 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33444: [udp sum ok] udp 12 (len 20, hlim 4) [tos 0x10] (ttl 64, id 47905, len 80)
Sep 12 09:07:58.157570 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33445: [udp sum ok] udp 12 (len 20, hlim 4) [tos 0x10] (ttl 64, id 13045, len 80)
^C


[kasperd]

Code Select Expand


# tcpdump -vvv -n -ttt -i sis2 'proto 41'
tcpdump: listening on sis2, link-type EN10MB
Sep 12 09:07:02.575764 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33435: [udp sum ok] udp 12 [hlim 1] (len 20) [tos 0x10] (ttl 64, id 2099, len 80)
Sep 12 09:07:07.578743 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33436: [udp sum ok] udp 12 [hlim 1] (len 20) [tos 0x10] (ttl 64, id 39099, len 80)
Sep 12 09:07:12.588564 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33437: [udp sum ok] udp 12 [hlim 1] (len 20) [tos 0x10] (ttl 64, id 30497, len 80)
Sep 12 09:07:17.599963 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33438: [udp sum ok] udp 12 (len 20, hlim 2) [tos 0x10] (ttl 64, id 16900, len 80)
Sep 12 09:07:22.608255 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33439: [udp sum ok] udp 12 (len 20, hlim 2) [tos 0x10] (ttl 64, id 38097, len 80)
Sep 12 09:07:27.618987 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33440: [udp sum ok] udp 12 (len 20, hlim 2) [tos 0x10] (ttl 64, id 64837, len 80)
Sep 12 09:07:38.119650 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33441: [udp sum ok] udp 12 (len 20, hlim 3) [tos 0x10] (ttl 64, id 55056, len 80)
Sep 12 09:07:43.127921 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33442: [udp sum ok] udp 12 (len 20, hlim 3) [tos 0x10] (ttl 64, id 33923, len 80)
Sep 12 09:07:48.137753 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33443: [udp sum ok] udp 12 (len 20, hlim 3) [tos 0x10] (ttl 64, id 63147, len 80)
Sep 12 09:07:53.149300 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33444: [udp sum ok] udp 12 (len 20, hlim 4) [tos 0x10] (ttl 64, id 47905, len 80)
Sep 12 09:07:58.157570 192.168.1.253 > 72.52.104.74: 2001:470:1f04:79d::2.28639 > 2001:470:1f04:79d::1.33445: [udp sum ok] udp 12 (len 20, hlim 4) [tos 0x10] (ttl 64, id 13045, len 80)
^C


Your tunnel endpoint is behind NAT. There is nothing wrong with that per se, but it opens up for lots of possible ways the connection can break. It depends very much on which NAT you are using, and how it is configured. On some NAT units it can work reliably, if properly configured. On some NAT units it doesn't work at all.

Try to run a tcpdump on the WAN interface of your NAT device.

[cam]

Thanks for the reply.
after rebooting my upstream device/router, the tunnel got  back to work.
That was indeed a tricky issue... Aztech to blame.. and me to trust that device to work flawlessly...

Will check it up and see if the problem can be reproduced and if rebooting that upstream device  solves the problem.

thanks !
Cheers,

[kasperd]

after rebooting my upstream device/router, the tunnel got  back to work.
That was indeed a tricky issue... Aztech to blame.

That can happen with other vendors too.