Tunnel stopped to work.

[derUhu]

Hallo! I have created an ipv6 tunnel and defined both routed/64 and routed/48 segments. I have set the tunnel up on RT-N56U router setting client IPV6 address to WAN port, allocated /64 subnet from routed segment and setting an address from it on LAN port. So far so good, everything was working perfectly. But then I decided to use only routed /64 subnet. I deleted routed /48, set LAN port to the address from routed/64 segment and after that tunnel stopped to work. I can ping both LAN and WAN addresses on my router from home LAN, but I cannot ping any address behind the tunnel, neither from PC nor from router itself (router allows to log in using ssh). I dropped and recreated the tunnel but nothing has changed. Is it possible to verify the tunnel configuretion on server side?

My tunnel parameters are:

Code Select Expand

Server IPv4 Address:  216.66.80.90
Server IPv6 Address:  2001:470:27:54e::1/64
Client IPv4 Address:  109.172.12.74
Client IPv6 Address:  2001:470:27:54e::2/64
Routed /64:           2001:470:28:54e::/64

Best regards,

[kasperd]

When I do a traceroute to your client IPv6 address it works fine all the way to the tunnel server, but I do not get a reply from your router. From that it appears the problem is on the tunnel, it could be either endpoint or a router on the connection between them dropping packets. For me the symptoms look the same regardless of whether I run the traceroute from a tunnel on the same tunnel server or from a tunnel on another tunnel server. I had no problem logging in on the host I have on the same tunnel server as you.

So this is most likely a problem affecting only your tunnel.

I tried bypassing the tunnel server and sending protocol 41 packets directly to your IPv4 address. I tested that with destination addresses in your tunnel prefix, your routed prefix as well as 6to4. In all three cases the packets are rejected by your router. I get ICMP errors of type 3 code 3 in response to the protocol 41 packets.

That means either 109.172.12.74 does not have a tunnel configured, or it is configured with filters that do not permit protocol 41 packets from my IP address.

It is possible your router is configured to only allow protocol 41 packets from one specific source IPv4 address. But given that the connection doesn't work at all for you, you should check the configuration of your router to verify that everything looks correct.

[derUhu]

I tried bypassing the tunnel server and sending protocol 41 packets directly to your IPv4 address. I tested that with destination addresses in your tunnel prefix, your routed prefix as well as 6to4. In all three cases the packets are rejected by your router. I get ICMP errors of type 3 code 3 in response to the protocol 41 packets.

That means either 109.172.12.74 does not have a tunnel configured, or it is configured with filters that do not permit protocol 41 packets from my IP address.

It is possible your router is configured to only allow protocol 41 packets from one specific source IPv4 address. But given that the connection doesn't work at all for you, you should check the configuration of your router to verify that everything looks correct.

109.172.12.74 is visible from outside as my fixed IP (and it is fixed, not dynamic) but actually it is NATted. The actual ip assigned to my router in in 10.* subnet.

On my router protocol 41 is accepted:

Code Select Expand

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination         
. . . . . . . . . . .
  106 13144 ACCEPT     41   --  *      *       0.0.0.0/0            0.0.0.0/0           
40148 2379K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

(OUTPUT chain just accepts all)

I do not know about provider's router: but the tunnel was working for about a week before...

On my router I have:

Code Select Expand

sit1      Link encap:IPv6-in-IPv4 
          inet6 addr: fe80::a4e:9106/128 Scope:Link
          inet6 addr: 2001:470:27:54e::2/64 Scope:Global
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13068 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:963690 (941.1 KiB)

Code Select Expand

/opt/home/admin # ip -6 route
2001:470:27:54e::/64 via :: dev sit1  proto kernel  metric 256
2001:470:28:54e::/64 dev br0  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
fe80::/64 dev eth3  proto kernel  metric 256
fe80::/64 via :: dev sit1  proto kernel  metric 256
default via 2001:470:27:54e::1 dev sit1  metric 1

and:

Code Select Expand

/opt/home/admin # ip link
. . . . . . . .
20: sit1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN
    link/sit 10.78.145.6 peer 216.66.80.90


[kasperd]

109.172.12.74 is visible from outside as my fixed IP (and it is fixed, not dynamic) but actually it is NATted. The actual ip assigned to my router in in 10.* subnet.

Ok. I repeated the traceroute with more visibility into the underlying IPv4 path enabled in my own gateway. This is the result

Code Select Expand

traceroute to 2001:470:27:54e::2 (2001:470:27:54e::2), 30 hops max, 80 byte packets
1  2a01:d0:839a:babe:735d:77a7:990d:702c  0.239 ms  0.126 ms  0.159 ms
2  ::ffff:10.0.0.1  0.585 ms  0.633 ms  0.648 ms
3  *  *  *
4  *  *  *
5  ::ffff:194.255.53.29  14.669 ms  24.081 ms  24.373 ms
6  2001:470:0:110::2  88.866 ms  89.180 ms  88.923 ms
7  *  *  *
8  *  *  *
9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  ::ffff:109.172.12.74  56.363 ms  58.991 ms  59.296 ms
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  ::ffff:109.172.12.74  60.374 ms !H  73.320 ms !H  55.376 ms !HAnd now let's compare this with a traceroute of the IPv4 path using an ordinary IPv4 traceroute. First run:

Code Select Expand

traceroute to 109.172.12.74 (109.172.12.74), 30 hops max, 60 byte packets
1  10.9.255.254  8.756 ms  9.188 ms  9.223 ms
2  212.10.10.1  9.446 ms  9.526 ms  9.609 ms
3  194.255.53.29  10.016 ms  10.480 ms  10.467 ms
4  * * *
5  213.248.66.145  11.794 ms  12.301 ms  12.287 ms
6  80.91.254.222  12.237 ms  10.407 ms  15.108 ms
7  213.248.65.165  24.909 ms  25.307 ms  24.521 ms
8  80.239.147.158  47.872 ms  48.267 ms  48.001 ms
9  62.115.11.190  58.830 ms  58.011 ms  57.568 ms
10  81.16.117.77  55.505 ms  55.938 ms  54.850 ms
11  92.246.130.94  54.734 ms  55.744 ms  60.362 ms
12  89.221.193.161  65.687 ms  74.491 ms  74.703 ms
13  89.221.199.129  92.879 ms  84.015 ms  83.625 ms
14  109.172.12.74  62.756 ms  62.999 ms  62.969 ms
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  109.172.12.74  65.735 ms  65.683 ms  64.993 msSecond run:

Code Select Expand

traceroute to 109.172.12.74 (109.172.12.74), 30 hops max, 60 byte packets
3  194.255.53.29  11.124 ms  11.461 ms  12.380 ms
4  213.248.66.146  13.646 ms  13.699 ms  13.782 ms
5  213.248.66.145  14.011 ms  14.039 ms  14.016 ms
6  80.91.254.222  13.315 ms  13.910 ms  17.287 ms
7  213.248.65.165  27.102 ms  27.370 ms  27.381 ms
8  80.239.147.158  50.602 ms  47.721 ms  47.806 ms
9  62.115.11.190  64.363 ms  63.668 ms  63.448 ms
10  81.16.117.77  119.116 ms  119.303 ms  118.999 ms
11  92.246.130.94  60.991 ms  60.988 ms  60.891 ms
12  89.221.193.161  63.163 ms  69.754 ms  73.098 ms
13  89.221.199.129  66.854 ms  55.322 ms  56.464 ms
14  109.172.12.74  56.754 ms  55.082 ms  61.866 ms
15  109.172.12.74  60.128 ms  66.374 ms  56.627 msEverything up until the first occurrence of 109.172.12.74 looks fine and consistent in all the traceroutes. Having two occurrences of 109.172.12.74 would be consistent with your explanation of the NAT layer. The missing five lines in some of the traceroutes are suspicious. Could mean ARP isn't working properly between the two routers. That is however unrelated to the problem, it just makes the other problem a bit harder to debug.

[derUhu]

I just tried to reset the firewall on my router (flushed all chains and verified that policies are ACCEPT everywhere). The tunnel still does not work :-/

[kasperd]

I have no idea why ICMP echo requests sometimes need six hops between those two routers and other times only one. Seeing a few extra hops and then a no route to host error would be perfectly normal if the no route to host error is caused by an ARP timeout. But this isn't the case here. Because after those extra hops, there is actually a reply.

A router responding to the ICMP echo request instead of sending a no route to host in the event of an ARP timeout would be possible, but a really strange thing to do. But people do strange things with NAT devices. However that cannot explain the symptoms either.

The reason an ARP timeout seems to produce the same number of extra hops each time is due to the fact that traceroute is sending the echo requests at some specific rate, and it simply shows how many more probes it can send before the ARP timeout.

But I tried varying the interval between echo requests by an order of magnitude, and the number of hops remained the same.

With protocol 41 packets those extra hops show up every time, but with ICMP echo requests they only show up some of the time.

Please show us what an IPv4 traceroute from your computer to 216.66.80.90 and to 109.172.12.74 look like.

[broquea]

Have you emailed ipv6@he.net and had them verify that the IPv4 endpoint is set correctly for your tunnel interface on their tserv?
If I leave my MTR running against 2001:470:27:54e::2, do you see the packets coming in via tcpdump?

[kasperd]

If you want to test that your own network is capable of tunnelling packets out and getting a reply back without involving the tunnel server, you are welcome to try to ping me.

Have your router use 86.52.121.5 as tunnel server and then try to ping 2001:470:1f0b:1da2:161c:67c3:678a:2837. It is not a real tunnel server, so not much will work, but you should be able to ping that one IPv6 address.

[snarked]

The only thing I see that's different from my set up is that on my SIT1 interface, I also have an IPv4 allocation assigned to it - the IPv4 address that is supposed to be the source address (and is my IPv4 destination endpoint) for my tunnelled packets.  I don't know if that will make a difference.  It shouldn't unless it's specifically filtered.  Addresses on real interfaces can be duplicated onto virtual ones.

[derUhu]

Traceroute from my router to tunnel server:

Code Select Expand

/opt/home/admin # traceroute -n 216.66.80.90
traceroute to 216.66.80.90 (216.66.80.90), 30 hops max, 38 byte packets
1  10.78.144.1  2.589 ms  3.223 ms  2.539 ms
2  172.30.78.90  21.601 ms  11.434 ms  4.334 ms
3  172.30.78.97  4.059 ms  6.366 ms  6.792 ms
4  172.31.78.3  1.252 ms  1.277 ms  1.129 ms
5  172.31.78.3  1.149 ms  1.540 ms  1.557 ms
6  172.30.78.137  1.075 ms  0.807 ms  0.805 ms
7  172.30.78.134  1.009 ms  0.921 ms  1.023 ms
8  89.221.199.129  1.511 ms  1.337 ms  1.530 ms
9  77.109.110.174  2.743 ms  2.673 ms  2.347 ms
10  77.109.110.173  1.913 ms  1.740 ms  2.149 ms
11  212.71.11.29  2.636 ms  1.957 ms  1.761 ms
12  212.71.11.177  14.194 ms  13.608 ms  13.644 ms
13  194.68.123.187  13.227 ms  17.920 ms  23.791 ms
14  216.66.80.90  13.331 ms  13.130 ms  13.104 ms
/opt/home/admin #

tracerouter from my router to it's external address looks as follows:

Code Select Expand

/opt/home/admin # traceroute -n 109.172.12.74
traceroute to 109.172.12.74 (109.172.12.74), 30 hops max, 38 byte packets
1  10.78.144.1  3.847 ms  3.785 ms  2.957 ms
2  172.30.78.90  5.752 ms  2.887 ms  3.079 ms
3  172.30.78.97  6.762 ms  16.390 ms  6.516 ms
4  172.31.78.3  1.415 ms  1.360 ms  1.333 ms
5  172.31.78.3  1.254 ms  1.131 ms  1.377 ms
6  172.30.78.137  0.838 ms  0.839 ms  0.828 ms
7  172.30.78.134  0.853 ms  0.943 ms  1.513 ms
8  89.221.199.129  1.259 ms  1.327 ms  1.467 ms
9  109.172.12.74  1.333 ms  1.331 ms  1.423 ms
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  109.172.12.74  2.333 ms  2.650 ms  2.741 ms

(just to remind that my router is NATted to 109.172.12.74 by provider)

And I tried to use 86.52.121.5 as tunnel server, but it still does not work.

[derUhu]

Sorry, ping to 2001:470:1f0b:1da2:161c:67c3:678a:2837 when using 86.52.121.5 as a tunnel server actually works!

Code Select Expand

/opt/home/admin # ping6 2001:470:1f0b:1da2:161c:67c3:678a:2837
PING 2001:470:1f0b:1da2:161c:67c3:678a:2837 (2001:470:1f0b:1da2:161c:67c3:678a:2837): 56 data bytes
64 bytes from 2001:470:1f0b:1da2:161c:67c3:678a:2837: seq=0 ttl=62 time=64.287 ms
64 bytes from 2001:470:1f0b:1da2:161c:67c3:678a:2837: seq=1 ttl=62 time=62.463 ms
64 bytes from 2001:470:1f0b:1da2:161c:67c3:678a:2837: seq=2 ttl=62 time=68.886 ms
64 bytes from 2001:470:1f0b:1da2:161c:67c3:678a:2837: seq=3 ttl=62 time=72.650 ms
64 bytes from 2001:470:1f0b:1da2:161c:67c3:678a:2837: seq=4 ttl=62 time=67.572 ms
^C
--- 2001:470:1f0b:1da2:161c:67c3:678a:2837 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 62.463/67.171/72.650 ms
/opt/home/admin #

[derUhu]

Ok, I just dropped the tunnel and created it with different server - and everything started to work again! Many thanks to all of you for you time and support!

[derUhu]

And the last piece of info: I dropped the new ly created tunnel and recreated the original one. Guess what? It works now... :-)

[broquea]

Sounds like the HE side didn't have the right IPv4 endpoint on the tserv set, if deleting and recreating fixed it.

[derUhu]

Deleting and recreating worked only after I used different tunnel server. When I just dropped the tunnel and immediately recreated it to the same tunnel server it did not work.