Problem IPV6

[campari]

root@Rashk0:~# ping6 2001:470:27:2ec::1
PING 2001:470:27:2ec::1(2001:470:27:2ec::1) 56 data bytes
^C
--- 2001:470:27:2ec::1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2016ms


root@Rashk0:~# ping6 2001:470:27:2ec::2
PING 2001:470:27:2ec::2(2001:470:27:2ec::2) 56 data bytes
64 bytes from 2001:470:27:2ec::2: icmp_seq=1 ttl=64 time=0.067 ms
64 bytes from 2001:470:27:2ec::2: icmp_seq=2 ttl=64 time=0.080 ms
^C
--- 2001:470:27:2ec::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.067/0.073/0.080/0.010 ms


root@Rashk0:~# host rashk0.no-ip.info
rashk0.no-ip.info has address 46.194.32.148



IPv6 Tunnel Endpoints
Server IPv4 Address:216.66.80.90
Server IPv6 Address:2001:470:27:2ec::1/64
Client IPv4 Address:46.194.32.148
Client IPv6 Address:2001:470:27:2ec::2/64

[cholzhauer]

You didn't explicitly say what the problem is, so I can only assume your tunnel isn't working.  You didn't give us much detail about your setup, so I can only assume you're behind NAT.

1) Did you use your NAT IP address when creating the tunnel?
2) Does your NAT appliance pass protocol41?
3) Is your public IP address correct on your tunnel page?

[campari]

You didn't explicitly say what the problem is, so I can only assume your tunnel isn't working.  You didn't give us much detail about your setup, so I can only assume you're behind NAT.

1) Did you use your NAT IP address when creating the tunnel?
2) Does your NAT appliance pass protocol41?
3) Is your public IP address correct on your tunnel page?

13: he-ipv6: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN
    link/sit 192.168.0.** peer 216.66.80.90

[cholzhauer]

That doesn't really tell me anything

I also see you had another thread about the same problem..did you ever follow the advice there?

[campari]

That doesn't really tell me anything

I also see you had another thread about the same problem..did you ever follow the advice there?

worked up until yesterday

[kasperd]

I tried to tunnel IPv6 packets from my network directly to your client IPv4 address to see if that would reveal anything about the whereabouts of the problem. I also compared the result with just running mtr towards your client IPv4 address.

The results were interesting:

Code Select Expand

                                       Packets               Pings
Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
1. 10.9.255.254                      0.0%    77   19.9  12.7   6.2  30.1   5.6
2. 212.10.10.20                      0.0%    77   12.3  13.5   6.3  39.3   7.1
3. 212.10.10.1                       0.0%    77   13.5  14.4   6.9  43.3   7.7
4. 194.255.53.29                     0.0%    77   11.3  14.5   7.4  36.7   7.6
5. 213.248.66.146                    1.3%    77   16.3  17.0   9.9  39.9   7.3
6. 213.248.66.145                    0.0%    77   10.6  18.8   9.6  70.8  13.2
7. 80.91.254.193                     0.0%    77   19.5  21.8  10.1 219.8  30.8
8. 213.248.73.178                    0.0%    77   13.0  21.4  10.4 115.6  20.4
9. 146.172.105.21                    0.0%    77   30.9  29.4  18.6  65.9  10.2
10. 146.172.102.157                   0.0%    77   31.6  26.8  18.4  54.3   8.0
11. 146.172.100.82                    0.0%    77   24.9  30.9  21.4  67.7  10.0
12. 146.172.110.158                   0.0%    77   19.8  28.4  19.3  65.5   8.3
13. 46.194.32.148                     0.0%    77   44.6  51.2  40.1  80.7   8.1

Code Select Expand

traceroute to 2001:470:27:2ec::2 (2001:470:27:2ec::2), 30 hops max, 80 byte packets
1  2a01:d0:839a:babe:735d:77a7:990d:702c  0.132 ms  0.114 ms  0.142 ms
2  ::ffff:10.0.0.1  0.298 ms  0.331 ms  0.359 ms
3  *  *  *
4  ::ffff:212.10.10.20  13.646 ms  14.020 ms  14.165 ms
5  *  *  *
6  ::ffff:194.255.53.29  14.702 ms  15.586 ms  16.421 ms
7  2001:470:20::2  90.950 ms  84.886 ms  87.058 ms
8  *  *  *
9  *  *  *
10  *  *  *
11  ::ffff:146.172.105.45  74.456 ms  19.928 ms  20.513 ms
12  ::ffff:146.172.99.177  21.525 ms  22.256 ms  21.695 ms
13  ::ffff:146.172.100.49  21.620 ms  21.664 ms  22.253 ms
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *In the IPv6 output hop number 7 is spurious. That's a router at my own ISP which cannot send TTL expired when the IPv4 protocol number is 41. Instead it simply removes the IPv4 header and forwards the IPv6 packet. That results in a spurious entry in the traceroute output, but that is unrelated to the problem.

The next three hops 8-10 are missing. I guess those are sending ICMP replies with insufficient data to recognize the original IPv6 header. Then we get to the part, which should be interesting to you. Hops number 11-13. Those should have corresponded to hops 9-11 in the IPv4 output. But they do not correspond, which suggests Telenor is routing protocol 41 packets in a different direction than ICMP packets.

I'd like you to try running traceroutes towards following IP addresses to see if that may reveal something useful:
194.255.53.29
146.172.105.45
146.172.100.49
146.172.105.21
146.172.110.158

Since something unexpected is happening inside the Telenor network. I am guessing the problem is there and not on the HE side. If your router supports setting up both 6in4 and 6to4 at the same time, then doing so and letting us know your IPv6 addresses from each of them would allow some more interesting traceroutes.

[kasperd]

I tried a traceroute using a 6to4 address as source address. It gave pretty much the same result. The spurious IP address at hop 7 is different, but that is of no significance.

Code Select Expand

traceroute to 2001:470:27:2ec::2 (2001:470:27:2ec::2), 30 hops max, 80 byte packets
1  2a01:d0:839a:babe:735d:77a7:990d:702c  0.148 ms  0.113 ms  0.137 ms
2  ::ffff:10.0.0.1  0.347 ms  0.380 ms  0.412 ms
3  *  *  *
4  ::ffff:212.10.10.20  41.612 ms  41.637 ms  41.617 ms
5  *  *  *
6  ::ffff:194.255.53.29  23.934 ms  23.957 ms  23.964 ms
7  2002:d842:505a::1  59.734 ms  43.773 ms  43.785 ms
8  *  *  *
9  *  *  *
10  *  *  *
11  ::ffff:146.172.105.45  28.128 ms  27.984 ms  27.999 ms
12  ::ffff:146.172.99.177  44.084 ms  44.092 ms  44.097 ms
13  ::ffff:146.172.100.49  43.997 ms  63.987 ms  64.002 ms
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *Some of the missing entries are due to ICMP packets with too short payload. Those tend to have 8 bytes of IPv4 payload. In case of protocol 41, those 8 bytes will be the first 8 bytes of the IPv6 header. Thus both source and destination IPv6 address are lost, and obviously the replies cannot be delivered to the traceroute command. I can still inspect them with tcpdump, but they won't contain much useful information. However the IPv6 TTL is still there, which is enough to reconstruct some of the missing hops, and here goes after manual reconstruction:

Code Select Expand

traceroute to 2001:470:27:2ec::2 (2001:470:27:2ec::2), 30 hops max, 80 byte packets
1  2a01:d0:839a:babe:735d:77a7:990d:702c  0.148 ms  0.113 ms  0.137 ms
2  ::ffff:10.0.0.1  0.347 ms  0.380 ms  0.412 ms
3  *  *  *
4  ::ffff:212.10.10.20  41.612 ms  41.637 ms  41.617 ms
5  *  *  *
6  ::ffff:194.255.53.29  23.934 ms  23.957 ms  23.964 ms
7  2002:d842:505a::1  59.734 ms  43.773 ms  43.785 ms
8  213.248.66.145
9  80.91.254.193
10  213.248.73.178
11  ::ffff:146.172.105.45  28.128 ms  27.984 ms  27.999 ms
12  ::ffff:146.172.99.177  44.084 ms  44.092 ms  44.097 ms
13  ::ffff:146.172.100.49  43.997 ms  63.987 ms  64.002 ms
14  146.172.9.74
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *Comparing this to the IPv4 mtr command, we can see that the packets take the same path until 213.248.73.178 and then the flows starts taking different paths. I'm not sure how many hops are from 146.172.9.74 to your IPv4 address.

[kasperd]

I tried traceroute with random port numbers to see if the difference could be due to some load balancing. But port numbers didn't seem to change the path the packets would take. However protocol numbers did change it. UDP, UDPlite and 6in4 packets take one path, TCP and ICMP take another path. Strange way to do load balancing.

[kasperd]

I don't think the two different routes is actually telling us anything about what the problem might be. It just makes it slightly harder to debug.

[kasperd]

Now I tried from a completely different network. Again I saw traceroute with ICMP taking a different route from protocol 41 packets.

mtr with ICMP packets:

Code Select Expand

                                       Packets               Pings
Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
1. 10.49.0.1                         0.0%     5   11.9  24.1   6.1  47.0  18.8
2. 10.250.10.1                       0.0%     5   30.7  18.0   8.9  30.7   7.9
3. 195.215.39.217                    0.0%     5  214.3  90.8  17.5 214.3  76.1
4. 88.131.143.118                    0.0%     5  114.8  50.0  14.2 114.8  40.3
5. 195.69.116.158                    0.0%     5   15.0  28.9  15.0  51.2  13.4
6. 146.172.105.105                   0.0%     4   93.9  35.0  15.0  93.9  39.3
7. 146.172.100.82                    0.0%     4   17.4  34.3  17.4  46.9  14.2
8. 146.172.110.158                   0.0%     4   35.6  36.1  21.0  44.0  10.7
9. 46.194.32.148                     0.0%     4   43.7  47.0  43.7  50.0   2.6

6to4:

Code Select Expand

traceroute to 2001:470:27:2ec::2 (2001:470:27:2ec::2), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.701 ms  0.809 ms  1.236 ms
2  ::ffff:10.0.0.1  1.826 ms  2.243 ms  2.296 ms
3  *  *  *
4  *  *  *
5  *  *  *
6  *  *  *
7  *  *  *
8  ::ffff:146.172.105.110  31.034 ms  132.381 ms  135.989 ms
9  ::ffff:146.172.100.49  19.936 ms  23.738 ms  27.782 ms
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *

And 6to4 with the gaps filled in for the short ICMP replies:

Code Select Expand

traceroute to 2001:470:27:2ec::2 (2001:470:27:2ec::2), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.701 ms  0.809 ms  1.236 ms
2  ::ffff:10.0.0.1  1.826 ms  2.243 ms  2.296 ms
3  10.49.0.1
4  10.250.10.1
5  195.215.39.217
6  88.131.143.118
7  195.69.116.158
8  ::ffff:146.172.105.110  31.034 ms  132.381 ms  135.989 ms
9  ::ffff:146.172.100.49  19.936 ms  23.738 ms  27.782 ms
10  146.172.9.74
11  *  *  *
12  *  *  *Even though the packets arrive from a different peer, the paths diverge immediately after entering the Telenor network.

[kasperd]

I know that protocol 41 packets I send to you go through 146.172.9.74. And if the two alternate paths have the same number of hops, then your client IPv4 would be the next hop on the path. And since that is the case for both the two source networks I tried from, I consider it a realistic guess, that the two alternate paths do indeed have the same number of hops.

I think that the protocol 41 packets, which I send did in fact reach your tunnel endpoint. If your tunnel endpoint was down or was rejecting the packets based on an unexpected source IPv4 address, there should have been an ICMP error send back to me. But I didn't get any ICMP errors from that hop. That means either something is dropping ICMP packets, or my packets did reach your tunnel endpoint.

Hence my hypothesis is that ICMPv6 echo replies did get send from your network towards me, but those never reached the tunnel server. To verify my hypothesis and to find anymore information, it is necessary to get some packet dumps from the physical interface with IPv4 address 46.194.32.148. Both protocol 41 and ICMP packets seen on that interface are of interest. It is unlikely any other protocols on the interface are of any interest.

And as mentioned before, if you can configure both 6to4 and 6in4 simultaneously on the router, it will make debugging much easier.

[campari]

I know that protocol 41 packets I send to you go through 146.172.9.74. And if the two alternate paths have the same number of hops, then your client IPv4 would be the next hop on the path. And since that is the case for both the two source networks I tried from, I consider it a realistic guess, that the two alternate paths do indeed have the same number of hops.

I think that the protocol 41 packets, which I send did in fact reach your tunnel endpoint. If your tunnel endpoint was down or was rejecting the packets based on an unexpected source IPv4 address, there should have been an ICMP error send back to me. But I didn't get any ICMP errors from that hop. That means either something is dropping ICMP packets, or my packets did reach your tunnel endpoint.

Hence my hypothesis is that ICMPv6 echo replies did get send from your network towards me, but those never reached the tunnel server. To verify my hypothesis and to find anymore information, it is necessary to get some packet dumps from the physical interface with IPv4 address 46.194.32.148. Both protocol 41 and ICMP packets seen on that interface are of interest. It is unlikely any other protocols on the interface are of any interest.

And as mentioned before, if you can configure both 6to4 and 6in4 simultaneously on the router, it will make debugging much easier.

41    TCP   192.168.0.63   41
is open
until yesterday was working ipv6  :(

[kasperd]

41    TCP   192.168.0.63   41
is open

Wrong protocol. It is not TCP you need to forward, it is protocol 41.

until yesterday was working ipv6

I don't know why it was ever working. Maybe you were relying on some fragile NAT handling of protocol 41, which just happened to break due to an unfortunate sequence of packets.

I am using a tunnel on the same tunnel server and haven't noticed even the slightest glitch. I don't think anything changed on the tunnel server. Most likely something changed at your ISP or on your own NAT. Your NAT is dropping most protocols without sending a proper reply back, which makes debugging from outside almost impossible.

Try to run a traceroute to each of these two IP addresses and run a packet dump on the interface the tunnel packets are being sent on.
2001:470:28:940:f3c0:44c3:77f:526f
2002:50a7:dea9:727a:2241:bd6c:dc87:7e5e

[kasperd]

I noticed that you did manage to get some packets through. So I tried pinging your IP address at increasing intervals to estimate any connection timeout in your NAT. Once the interval between pings reached one week, your IP stopped responding. This may be a coincidence. A one week timeout sounds too high to be likely, and it is also unlikely that there would be no other traffic for such a long time.