Quality of Service breaks IPv6 6in4 tunnel.

[kr1zmo]

I am using a 6in4 tunnel for IPv6 support, everything works fine until I enable Quality of Service. QOS seems to break IPv6 support. I can ping the ipv6 router address, but cannot reach anything outside of it - dns server, ips, etc)

I am using Asus RT-N56u Firmware Version:3.0.0.4.360 and setting-up its 6in4 tunnel support built in the router.

Do you think it has to do with QOS affecting the IPv4 encapsulation over protocol 41?

Is there anyway that since the tunnel uses IPv4 for transmission the QOS is messing up the packets.

Code Select Expand


---- ROUTER PING ---
PING6(56=40+8+8 bytes) 2001:470:8:6df:xxxx:47ff:fe07:b2ea --> 2001:470:8:xxx::1
16 bytes from 2001:470:8:6df::1, icmp_seq=0 hlim=64 time=1.057 ms
16 bytes from 2001:470:8:6df::1, icmp_seq=1 hlim=64 time=0.923 ms
16 bytes from 2001:470:8:6df::1, icmp_seq=2 hlim=64 time=0.933 ms
16 bytes from 2001:470:8:6df::1, icmp_seq=3 hlim=64 time=0.892 ms
^C
--- 2001:470:8:6df::1 ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.892/0.951/1.057/0.063 ms

--- PING GOOGLE IPV6 ---
Joshua@MacBookAir ~> ping6 ipv6.google.com
PING6(56=40+8+8 bytes) 2001:470:8:6df:c8bb:840b:xxxx:6712 --> 2607:f8b0:4004:801::1010
^C
--- ipv6.l.google.com ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss



All IPv6 connections timeout when QOS is on?

Thanks.

[kasperd]

Why would changing the QoS settings also change the source IPv6 address on ICMPv6 hop limit exceeded messages from the router? It does sound more likely that the difference is caused by some other setting, which you accidentally changed at the same time.

It used to use 2002:422a:d452:: as source address, but is now using 2001:470:7:6e0::2.

That's not to say that changing QoS settings couldn't break protocol 41. But it is more likely such a breakage would happen in the ISP routers doing something stupid with such packets. That however doesn't explain the change in IP address.

For the record I don't think the HE DNS servers could be responsible for the problem.

Here is what a traceroute looks like from my end now:

Code Select Expand

traceroute to 2001:470:8:6df:62c5:47ff:fe07:xxxx (2001:470:8:6df:62c5:47ff:fe07:b2ea), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.465 ms  0.464 ms  0.648 ms
2  2001:470:27:940::1  39.529 ms  44.578 ms  47.867 ms
3  2001:470:0:11e::1  48.610 ms  34.666 ms  45.874 ms
4  2001:470:0:22f::1  70.130 ms  70.432 ms  67.243 ms
5  2001:470:0:3f::1  82.254 ms  82.888 ms  83.124 ms
6  2001:470:0:128::1  141.848 ms  199.135 ms  188.217 ms
7  2001:470:0:299::1  202.242 ms  169.814 ms  174.568 ms
8  2001:470:0:90::2  174.055 ms  177.593 ms  177.690 ms
9  2001:470:7:6e0::2  203.933 ms  193.242 ms  195.206 ms
10  2001:470:8:6df:62c5:47ff:fe07:xxxx  221.109 ms  223.337 ms  225.324 ms

To investigate further I suggest you try changing the setting a few more times, and between each change run a traceroute6 from outside your network to an IP address inside the network to confirm that this one setting is responsible for both changes in behaviour.

[kr1zmo]

Before QOS enabled:

Code Select Expand

TraceRoute IPv6 Output:
traceroute to 2001:470:8:6df:3c24:517:a33b:8e9c (2001:470:8:6df:3c24:517:a33b:8e9c), 30 hops max, 40 byte packets
1  2a02:348:82::1 (2a02:348:82::1)  10.612 ms  10.747 ms  10.863 ms
2  xl-internetservices.nl.ip6.jointtransit.nl (2a02:10:0:1::e:3)  10.686 ms  10.569 ms  10.831 ms
3  hurricane-electric.nikhef.nlsix.net (2001:7f8:13::a500:6939:1)  10.911 ms  10.914 ms  10.922 ms
4  10gigabitethernet1-4.core1.lon1.he.net (2001:470:0:3f::1)  12.817 ms  12.740 ms 10gigabitethernet2-1.core1.par2.he.net (2001:470:0:1b3::2)  11.483 ms
5  10gigabitethernet15-1.core1.ash1.he.net (2001:470:0:1b1::1)  85.342 ms  89.337 ms 10gigabitethernet10-4.core1.nyc4.he.net (2001:470:0:128::1)  86.215 ms
6  tserv2.ash1.he.net (2001:470:0:90::2)  86.576 ms  86.903 ms  86.756 ms
7  tserv2.ash1.he.net (2001:470:0:90::2)  82.001 ms  85.042 ms  83.407 ms
8  kr1zmo-1-pt.tunnel.tserv13.ash1.ipv6.he.net (2001:470:7:6e0::2)  109.447 ms  111.527 ms  (2001:470:8:6df:3c24:517:a33b:8e9c)  120.276 ms


Finished!

After QOS enabled:

Code Select Expand

TraceRoute IPv6 Output:
traceroute to 2001:470:8:6df:3c24:517:a33b:8e9c (2001:470:8:6df:3c24:517:a33b:8e9c), 30 hops max, 40 byte packets
1  2a02:348:82::1 (2a02:348:82::1)  8.810 ms  8.948 ms  9.076 ms
2  xl-internetservices.nl.ip6.jointtransit.nl (2a02:10:0:1::e:3)  8.821 ms  9.192 ms  9.282 ms
3  hurricane-electric.nikhef.nlsix.net (2001:7f8:13::a500:6939:1)  9.018 ms  9.016 ms  9.020 ms
4  10gigabitethernet2-1.core1.par2.he.net (2001:470:0:1b3::2)  11.490 ms 10gigabitethernet1-4.core1.lon1.he.net (2001:470:0:3f::1)  9.045 ms 10gigabitethernet2-1.core1.par2.he.net (2001:470:0:1b3::2)  11.412 ms
5  10gigabitethernet10-4.core1.nyc4.he.net (2001:470:0:128::1)  80.055 ms 10gigabitethernet15-1.core1.ash1.he.net (2001:470:0:1b1::1)  86.132 ms  85.295 ms
6  100gigabitethernet11-1.core1.ash1.he.net (2001:470:0:299::1)  80.712 ms  87.279 ms tserv2.ash1.he.net (2001:470:0:90::2)  87.477 ms
7  * tserv2.ash1.he.net (2001:470:0:90::2)  84.027 ms *
8  * * *
9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

---- Finished ------


All QOS settings were default and set by manufacturer firmware.
I can ping - from the router - my ipv4 tunnel address.
My computers shows a valid ipv6 address.
Everything works fine until I enable QOS - I've done this probably 10 times - same result.
I can ping6 my internal devices such as my debian server and ssh, I can also ping6 my router ipv6.
It seems I just cannot access the outside network over ipv6; it has to do something with QOS affecting ipv6 encapsulated packets on ipv4 protocol 41.

[kasperd]

Everything works fine until I enable QOS - I've done this probably 10 times - same result.
I can ping6 my internal devices such as my debian server and ssh, I can also ping6 my router ipv6.
It seems I just cannot access the outside network over ipv6; it has to do something with QOS affecting ipv6 encapsulated packets on ipv4 protocol 41.

We need to find out if this problem is caused by the router or by the ISP. We also need to find out if it is affecting packets in both directions or only in one direction.

Can you try to ping these two addresses with QoS enabled. Do not try to ping them with QoS disabled. Then I can see if you can send any outgoing packets.

2001:470:28:940:c809:a034:cc49:1244
2002:50a7:dea9:727a:697f:74ad:ae3c:5da

Do you know how to get packet dumps from the WAN interface on the router? That would be the best way to find out if the problem is caused by the router or the ISP.

[kr1zmo]

On both, ping failed:

A traceroute:

Code Select Expand

traceroute to 2001:470:28:940:c809:a034:cc49:1244 (2001:470:28:940:c809:a034:cc49:1244), 30 hops max, 80 byte packets
1  2001:470:8:6df::1 (2001:470:8:6df::1)  0.453 ms  0.455 ms  0.612 ms
2  * * *
3  * * *
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *
11  * * *
12  * *^C

I am not sure how to grab packets that are lost, I have shell (telnet) access to my router, can I do it there?

In my routing table I noticed this:

Code Select Expand

2002:50a7:dea9:727a:697f:74ad:ae3c:5da/128  2002:50a7:dea9:727a:697f:74ad:ae3c:5da  UC    0      173     170 v6in4  

Code Select Expand

IPv6 routing table
Destination                                 Next Hop                                Flags Metric Ref    Use Iface
2001:470:7:6e0::/64                         ::                                      U     256    0        0 v6in4  
2001:470:8:6df::/64                         ::                                      U     256    0        0 br0    
2001:470:28:940:c809:a034:cc49:1244/128     2001:470:28:940:c809:a034:cc49:1244     UC    0      76      70 v6in4  
2001:67c:2b0::6/128                         2001:67c:2b0::6                         UC    0      2        2 v6in4  
2002:50a7:dea9:727a:697f:74ad:ae3c:5da/128  2002:50a7:dea9:727a:697f:74ad:ae3c:5da  UC    0      173     170 v6in4  
2606:2800:220:bf1:95:a65:51f:1a94/128       2606:2800:220:bf1:95:a65:51f:1a94       UC    0      4        4 v6in4  
2607:f8b0:4004:800::1000/128                2607:f8b0:4004:800::1000                UC    0      6        6 v6in4  
2607:f8b0:4004:800::1001/128                2607:f8b0:4004:800::1001                UC    0      2        2 v6in4  
2607:f8b0:4004:800::1002/128                2607:f8b0:4004:800::1002                UC    0      2        2 v6in4  
2607:f8b0:4004:800::1003/128                2607:f8b0:4004:800::1003                UC    0      2        2 v6in4  
2607:f8b0:4004:800::1006/128                2607:f8b0:4004:800::1006                UC    0      1        1 v6in4  
2607:f8b0:4004:800::1007/128                2607:f8b0:4004:800::1007                UC    0      12       4 v6in4  
2607:f8b0:4004:800::1008/128                2607:f8b0:4004:800::1008                UC    0      1        1 v6in4  
2607:f8b0:4004:800::100a/128                2607:f8b0:4004:800::100a                UC    0      1        1 v6in4  
2607:f8b0:4004:800::100b/128                2607:f8b0:4004:800::100b                UC    0      6        6 v6in4  
2607:f8b0:4004:800::100e/128                2607:f8b0:4004:800::100e                UC    0      2        2 v6in4  
2607:f8b0:4004:800::100f/128                2607:f8b0:4004:800::100f                UC    0      8        8 v6in4  
2607:f8b0:4004:800::1012/128                2607:f8b0:4004:800::1012                UC    0      5        5 v6in4  
2607:f8b0:4004:800::1013/128                2607:f8b0:4004:800::1013                UC    0      5        5 v6in4  
2607:f8b0:400c:c02::7d/128                  2607:f8b0:400c:c02::7d                  UC    0      28      18 v6in4  
2610:148:1f10:3::89/128                     2610:148:1f10:3::89                     UC    0      3        3 v6in4  
2a00:1450:4008:c01::7d/128                  2a00:1450:4008:c01::7d                  UC    0      15       5 v6in4  
2a03:7900:2:0:31:3:104:56/128               2a03:7900:2:0:31:3:104:56               UC    0      2        2 v6in4  
2a03:7900:64::25f:33dc/128                  2a03:7900:64::25f:33dc                  UC    0      6        2 v6in4  
2a03:7900:64::3cad:1ae8/128                 2a03:7900:64::3cad:1ae8                 UC    0      6        2 v6in4  
2a03:7900:64::403e:f37a/128                 2a03:7900:64::403e:f37a                 UC    0      6        2 v6in4  
2a03:7900:64::42c5:b44a/128                 2a03:7900:64::42c5:b44a                 UC    0      3        1 v6in4  
2a03:7900:64::5f9e:fa55/128                 2a03:7900:64::5f9e:fa55                 UC    0      6        2 v6in4  
fe80::/64                                   ::                                      U     256    0        0 eth2    
fe80::/64                                   ::                                      U     256    0        0 ra0    
fe80::/64                                   ::                                      U     256    0        0 rai0    
fe80::/64                                   ::                                      U     256    0        0 br0    
fe80::/64                                   ::                                      U     256    0        0 wdsi0  
fe80::/64                                   ::                                      U     256    0        0 wdsi1  
fe80::/64                                   ::                                      U     256    0        0 wdsi2  
fe80::/64                                   ::                                      U     256    0        0 wdsi3  
fe80::/64                                   ::                                      U     256    0        0 wds0    
fe80::/64                                   ::                                      U     256    0        0 wds1    
fe80::/64                                   ::                                      U     256    0        0 wds2    
fe80::/64                                   ::                                      U     256    0        0 wds3    
fe80::/64                                   ::                                      U     256    0        0 eth3    
fe80::/64                                   ::                                      U     256    0        0 v6in4  
::/0                                        ::                                      U     1      0        0 v6in4  
::1/128                                     ::                                      U     0      0        1 lo      
2001:470:7:6e0::/128                        ::                                      U     0      0        2 lo      
2001:470:7:6e0::2/128                       ::                                      U     0      0        1 lo      
2001:470:8:6df::/128                        ::                                      U     0      0        2 lo      
2001:470:8:6df::1/128                       ::                                      U     0      2        1 lo      
fe80::/128                                  ::                                      U     0      0        2 lo      
fe80::/128                                  ::                                      U     0      0        2 lo      
fe80::/128                                  ::                                      U     0      0        2 lo      
fe80::/128                                  ::                                      U     0      0        2 lo      
fe80::/128                                  ::                                      U     0      0        2 lo      
fe80::422a:d452/128                         ::                                      U     0      0        1 lo      
fe80::beae:c5ff:feeb:fba5/128               ::                                      U     0      0        1 lo      
fe80::beae:c5ff:feeb:fba6/128               ::                                      U     0      0        1 lo      
fe80::beae:c5ff:feeb:fba6/128               ::                                      U     0      0        1 lo      
fe80::beae:c5ff:feeb:fba6/128               ::                                      U     0      0        1 lo      
fe80::beae:c5ff:feeb:fba6/128               ::                                      U     0      27       1 lo      
ff02::1/128                                 ff02::1                                 UC    0      104       0 br0    
ff00::/8                                    ::                                      U     256    0        0 eth2    
ff00::/8                                    ::                                      U     256    0        0 ra0    
ff00::/8                                    ::                                      U     256    0        0 rai0    
ff00::/8                                    ::                                      U     256    0        0 br0    
ff00::/8                                    ::                                      U     256    0        0 wdsi0  
ff00::/8                                    ::                                      U     256    0        0 wdsi1  
ff00::/8                                    ::                                      U     256    0        0 wdsi2  
ff00::/8                                    ::                                      U     256    0        0 wdsi3  
ff00::/8                                    ::                                      U     256    0        0 wds0    
ff00::/8                                    ::                                      U     256    0        0 wds1    
ff00::/8                                    ::                                      U     256    0        0 wds2    
ff00::/8                                    ::                                      U     256    0        0 wds3    
ff00::/8                                    ::                                      U     256    0        0 eth3    
ff00::/8                                    ::                                      U     256    0        0 v6in4

[kr1zmo]

That would be the best way to find out if the problem is caused by the router or the ISP.

Why could IPv6 connectivity loss after enabling QOS be from the ISP? QOS just distributes connections between hosts on the local network by balancing out connection priorities right? How could the ISP affect this? I can tell you that IPv4 with QOS works perfectly.

[qbot]

this looks a lot like this windows 8/kaspersky/icmp problem on dslreports http://www.dslreports.com/forum/r28292303-Windows-8-ipv6-issues-TunnelBroker-HE- , protocol 41 is in IPv4 so no issue there.  But if enabled QoS causes packet to be built in way that requires router inspection of packet would explain why tracert looks just like one in dslreports.  maybe try disabling any 3rd party AV software, and test non-ICMP connections.

[kasperd]

On both, ping failed:

A traceroute:

Code Select Expand

traceroute to 2001:470:28:940:c809:a034:cc49:1244 (2001:470:28:940:c809:a034:cc49:1244), 30 hops max, 80 byte packets
1  2001:470:8:6df::1 (2001:470:8:6df::1)  0.453 ms  0.455 ms  0.612 ms
2  * * *


None of those packets made it to me. So it appears you do have a problem on the outgoing path. We don't yet know, if you also have a problem on the incoming path.

I am not sure how to grab packets that are lost, I have shell (telnet) access to my router, can I do it there?

You can telnet to the router and run tcpdump. For a starter try tcpdump -pni any 'proto 41 || icmp'

Why could IPv6 connectivity loss after enabling QOS be from the ISP? QOS just distributes connections between hosts on the local network by balancing out connection priorities right?

The router might also set ToS bits in the IPv4 header. And the ISP might be mishandling that particular combination of ToS bits and protocol version. I don't know if that is what happened, but it is one possibility to investigate.

[kasperd]

this looks a lot like this windows 8/kaspersky/icmp problem on dslreports http://www.dslreports.com/forum/r28292303-Windows-8-ipv6-issues-TunnelBroker-HE-

I looked through that thread. It sounds like Windows 8 is including a hop-by-hop extension header with router alerts on all ping and traceroute packets. I didn't get the Kaspersky connection, if there even was one. There was also some guessing about Teredo issues, but that may be a red herring.

I have a test page on http://test-ipv6.netiter.net/ which could test for the Teredo issue (sorry, I didn't get around to translate the page yet. But just mention the test-ID here, and I'll tell you what the test found). It might turn out your problem is one, which that test page won't say much about.

As for the hop-by-hop header, I can try to reproduce that problem another day. I don't have time to hack together the code needed for that just now.

[kr1zmo]

At this point I'm not sure what to do. My new issue is, after some idle time, the tunnel does down completely. I still have a valid IPv6 address on my device, but I have no connection. I have read that others are experiencing this and have to write scripts that ping to keep the tunnel open. At night everything is fine, but I sleep and wake up and the IPv6 is down.

About to give up.

[kasperd]

At this point I'm not sure what to do. My new issue is, after some idle time, the tunnel does down completely. I still have a valid IPv6 address on my device, but I have no connection. I have read that others are experiencing this and have to write scripts that ping to keep the tunnel open. At night everything is fine, but I sleep and wake up and the IPv6 is down.

Two issues that may be related to this. One possibility is your IPv4 address is dynamic, in which case you need to tell the tunnelbroker service, when it has changed. Another possibility is that your tunnel endpoint is behind a NAT, and your tunnel through the NAT times out. Usually the NAT issue only breaks connections from the outside in to your tunnel endpoint. Packets going out will still work and reestablish the tunnel, which will then work for incoming connections as well, until it times out again.

I have a script, which can probe an address at increasing intervals to estimate the NAT timeout. I have started that script pinging the IPv6 address mentioned above, I'll let you know what results it comes up with.

[kasperd]

After QOS enabled:

Code Select Expand

TraceRoute IPv6 Output:
traceroute to 2001:470:8:6df:3c24:517:a33b:8e9c (2001:470:8:6df:3c24:517:a33b:8e9c), 30 hops max, 40 byte packets
1  2a02:348:82::1 (2a02:348:82::1)  8.810 ms  8.948 ms  9.076 ms
2  xl-internetservices.nl.ip6.jointtransit.nl (2a02:10:0:1::e:3)  8.821 ms  9.192 ms  9.282 ms
3  hurricane-electric.nikhef.nlsix.net (2001:7f8:13::a500:6939:1)  9.018 ms  9.016 ms  9.020 ms
4  10gigabitethernet2-1.core1.par2.he.net (2001:470:0:1b3::2)  11.490 ms 10gigabitethernet1-4.core1.lon1.he.net (2001:470:0:3f::1)  9.045 ms 10gigabitethernet2-1.core1.par2.he.net (2001:470:0:1b3::2)  11.412 ms
5  10gigabitethernet10-4.core1.nyc4.he.net (2001:470:0:128::1)  80.055 ms 10gigabitethernet15-1.core1.ash1.he.net (2001:470:0:1b1::1)  86.132 ms  85.295 ms
6  100gigabitethernet11-1.core1.ash1.he.net (2001:470:0:299::1)  80.712 ms  87.279 ms tserv2.ash1.he.net (2001:470:0:90::2)  87.477 ms
7  * tserv2.ash1.he.net (2001:470:0:90::2)  84.027 ms *
8  * * *
9  * * *
10  * * *


Where did you get that output from. I cannot reproduce it. I get this

Code Select Expand

traceroute to 2001:470:8:6df:3c24:517:a33b:8e9c (2001:470:8:6df:3c24:517:a33b:8e9c), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.404 ms  1.587 ms  5.128 ms
2  2001:470:27:940::1  65.233 ms  70.790 ms  69.458 ms
3  2001:470:0:11e::1  75.247 ms  34.069 ms  39.195 ms
4  2001:470:0:22f::1  69.482 ms  70.679 ms  117.503 ms
5  2001:470:0:1b3::2  129.379 ms  124.851 ms  126.800 ms
6  2001:470:0:1b1::1  196.318 ms  189.911 ms  192.601 ms
7  2001:470:0:90::2  194.172 ms  160.831 ms  162.144 ms
8  2002:422a:d452::  185.037 ms  190.151 ms  255.862 ms
9  *  *  *
10  *  *  *Notice that in the output you provided, there is no response from your router. In the output I got, your router is sending a reply with source IP 2002:422a:d452::

[kr1zmo]

Thanks. I'll reply back later today or tomorrow, I have work.

[kasperd]

I just noticed I still had a ping running, no replies from your IP for a week. I'll shut it down now.