Tunnels stopped working a few days ago (dont know when exactly)

[mebbrecht]

Hi folks,

I got a strange Problem with three of my tunnels. A few days ago the stopped working:

One Tunnel as example:
System is Debian Squeeze, Berlin DE is endpoint
Linux storage1 2.6.32-5-xen-686 #1 SMP Wed Apr 9 23:09:56 UTC 2014 i686 GNU/Linux (The Problem also appears on another vm and physical machine).

/etc/network/interfaces:

auto eth4
iface eth4 inet static
  address 1.2.3.100
  netmask 255.255.255.248
  gateway 1.2.3.97

auto sit1
iface sit1 inet6 v4tunnel
    address 2001:123:6c:111::2
    netmask 64
    local 1.2.3.100
    endpoint 216.66.86.114
    ttl 255
    up ip addr add 2001:123:6d:111:216:3eff:fe4e:8082/64 dev sit1
    up ip route add 2000::0/3 via 2001:123:6c:111::1 dev sit1

ifconfig:
[...]
eth4      Link encap:Ethernet  Hardware Adresse 00:16:3e:03:2b:e9
          inet Adresse:1.2.3.100  Bcast:1.2.3.103  Maske:255.255.255.248
          inet6-Adresse: fe80::216:3eff:fe03:2be9/64 Gültigkeitsbereich:Verbindung
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metrik:1
          RX packets:65926 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3082 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:1000
          RX bytes:6794447 (6.4 MiB)  TX bytes:827626 (808.2 KiB)
          Interrupt:235
[...]
sit1      Link encap:IPv6-nach-IPv4
          inet6-Adresse: 2001:123:6d:111:216:3eff:fe4e:8082/64 Gültigkeitsbereich:Global
          inet6-Adresse: 2001:123:6c:111::2/64 Gültigkeitsbereich:Global
          inet6-Adresse: fe80::5c32:4264/128 Gültigkeitsbereich:Verbindung
          UP PUNKTZUPUNKT RUNNING NOARP  MTU:1480  Metrik:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:0
          RX bytes:0 (0.0 B)  TX bytes:1664 (1.6 KiB)

route -6:
2001:123:6c:111::/64           ::                         Un   256 0     1 sit1
2001:123:6d:111::/64           ::                         Un   256 0     0 sit1
2000::/3                       2001:123:6c:111::1         UG   1024 0    19 sit1
fe80::/64                      ::                         U    256 0     0 eth0
fe80::/64                      ::                         U    256 0     0 eth1
fe80::/64                      ::                         U    256 0     0 eth2
fe80::/64                      ::                         U    256 0     0 eth3
fe80::/64                      ::                         U    256 0     0 eth4
fe80::/64                      ::                         U    256 0     0 eth5
fe80::/64                      ::                         Un   256 0     0 sit1
::/0                           ::                         !n   -1  1   629 lo
::1/128                        ::                         Un   0   1   267 lo
2001:123:6c:111::2/128         ::                         Un   0   1     0 lo
2001:123:6d:111:216:3eff:fe4e:8082/128 ::                         Un   0   1     0 lo
fe80::5c32:4264/128            ::                         Un   0   1     0 lo
fe80::216:3eff:fe03:2be9/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe20:7bc9/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe2f:ab56/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe3e:f100/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe42:121f/128   ::                         Un   0   1    90 lo
fe80::216:3eff:fe4e:8082/128   ::                         Un   0   1     0 lo
ff00::/8                       ::                         U    256 0     0 eth0
ff00::/8                       ::                         U    256 0     0 eth1
ff00::/8                       ::                         U    256 0     0 eth2
ff00::/8                       ::                         U    256 0     0 eth3
ff00::/8                       ::                         U    256 0     0 eth4
ff00::/8                       ::                         U    256 0     0 eth5
ff00::/8                       ::                         U    256 0     0 sit1
::/0                           ::                         !n   -1  1   629 lo

Ping:

ping6 2001:123:6c:111::1
PING 2001:123:6c:111::1(2001:123:6c:111::1) 56 data bytes
^C
--- 2001:123:6c:111::1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2013ms


ping6 -n heise.de
PING heise.de(2a02:2e0:3fe:1001:302::) 56 data bytes
^C
--- heise.de ping statistics ---
12 packets transmitted, 0 received, 100% packet loss, time 11087ms

IPTables is POLICY ACCEPT - trust me ;)

I created a new Tunnel (Frankfurt, DE is endpoint):

/etc/network/interfaces:

auto eth4
iface eth4 inet static
  address 1.2.3.101
  netmask 255.255.255.248
  gateway 1.2.3.97

auto sit1
iface sit1 inet6 v4tunnel
    address 2001:132:1f0a:1c8::2
    netmask 64
    local 1.2.3.101
    endpoint 216.66.80.30
    ttl 255
    up ip addr add 2001:132:1f0b:1c8:216:3eff:fe4e:8082/64 dev sit1
    up ip route add 2000::0/3 via 2001:470:1f0a:1c8::1 dev sit1

ifconfig:

[...]
eth4      Link encap:Ethernet  Hardware Adresse 00:16:3e:03:2b:e9
          inet Adresse:1.2.3.101  Bcast:1.2.3.103  Maske:255.255.255.248
          inet6-Adresse: fe80::216:3eff:fe03:2be9/64 Gültigkeitsbereich:Verbindung
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metrik:1
          RX packets:36 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:1000
          RX bytes:2828 (2.7 KiB)  TX bytes:1506 (1.4 KiB)
          Interrupt:235
[...]

sit1      Link encap:IPv6-nach-IPv4
          inet6-Adresse: 2001:123:1f0b:1c8:216:3eff:fe4e:8082/64 Gültigkeitsbereich:Global
          inet6-Adresse: 2001:123:1f0a:1c8::2/64 Gültigkeitsbereich:Global
          inet6-Adresse: fe80::5c32:4265/128 Gültigkeitsbereich:Verbindung
          UP PUNKTZUPUNKT RUNNING NOARP  MTU:1480  Metrik:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:0
          RX bytes:208 (208.0 B)  TX bytes:208 (208.0 B)

route -6:
2001:123:1f0a:1c8::/64         ::                         Un   256 0     1 sit1
2001:123:1f0b:1c8::/64         ::                         Un   256 0     0 sit1
2000::/3                       2001:123:1f0a:1c8::1       UG   1024 0     3 sit1
fe80::/64                      ::                         U    256 0     0 eth0
fe80::/64                      ::                         U    256 0     0 eth1
fe80::/64                      ::                         U    256 0     0 eth2
fe80::/64                      ::                         U    256 0     0 eth3
fe80::/64                      ::                         U    256 0     0 eth4
fe80::/64                      ::                         U    256 0     0 eth5
fe80::/64                      ::                         Un   256 0     0 sit1
::/0                           ::                         !n   -1  1    25 lo
::1/128                        ::                         Un   0   1    21 lo
2001:123:1f0a:1c8::2/128       ::                         Un   0   1     0 lo
2001:123:1f0b:1c8:216:3eff:fe4e:8082/128 ::                         Un   0   1     2 lo
fe80::5c32:4265/128            ::                         Un   0   1     0 lo
fe80::216:3eff:fe03:2be9/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe20:7bc9/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe2f:ab56/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe3e:f100/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe42:121f/128   ::                         Un   0   1     0 lo
fe80::216:3eff:fe4e:8082/128   ::                         Un   0   1     0 lo
ff00::/8                       ::                         U    256 0     0 eth0
ff00::/8                       ::                         U    256 0     0 eth1
ff00::/8                       ::                         U    256 0     0 eth2
ff00::/8                       ::                         U    256 0     0 eth3
ff00::/8                       ::                         U    256 0     0 eth4
ff00::/8                       ::                         U    256 0     0 eth5
ff00::/8                       ::                         U    256 0     0 sit1
::/0                           ::                         !n   -1  1    25 lo

Ping6:

ping6 2001:123:1f0a:1c8::1
PING 2001:123:1f0a:1c8::1(2001:123:1f0a:1c8::1) 56 data bytes
64 bytes from 2001:123:1f0a:1c8::1: icmp_seq=1 ttl=64 time=16.9 ms
64 bytes from 2001:123:1f0a:1c8::1: icmp_seq=2 ttl=64 time=23.8 ms
^C
--- 2001:123:1f0a:1c8::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 16.994/20.414/23.835/3.423 ms

ping6 -n heise.de
PING heise.de(2a02:2e0:3fe:1001:302::) 56 data bytes
64 bytes from 2a02:2e0:3fe:1001:302::: icmp_seq=1 ttl=59 time=44.1 ms
64 bytes from 2a02:2e0:3fe:1001:302::: icmp_seq=2 ttl=59 time=22.0 ms
64 bytes from 2a02:2e0:3fe:1001:302::: icmp_seq=3 ttl=59 time=14.4 ms
^C
--- heise.de ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.440/26.866/44.116/12.587 ms

--------------------------------------

I got 5 Tunnels:
1st - Connected via Fritzbox to Berlin - working - created OCT2013 - dynamic ip
2nd - Connected via Linux to Berlin - not working - created OCT2013 - static ip
3rd - Connected via Linux to Berlin - not working - created OCT2013 - static ip
4th - Connected via Linux to Berlin - not working - created OCT2013 - static ip (the not working one described above)
5th - Connected via Linux to Frankfurt - working (the new one described above) - static ip

Got DNS Entries for 2,3,4 and a /48 on the 2nd (and use this in several networks behind the gateway) so dont wanna recreate them (much much work). Support is involved but not responding at the moment :(

The tunnels worked a few days ago ... so my questions:
1. Has anyone experienced the same problems (with Berlin) ?
2. Any hints on revealing or resolving this problem
3. Could it be, that the tunnel is just broken on server side ?

Thanks in advance :)

Marcel

[mebbrecht]

Got an answer:

Those Linux systems are seemingly setting QoS parameters on the
underlying IPv4 traffic, which is throwing warnings on this side.

In any case, I checked on XXXXXX to start and see the following traffic:
04:45:00.878671 IP (tos 0x3,CE, ttl 244, id 0, offset 0, flags [DF],
proto IPv6 (41), length 100)
    1.2.3.98 > 216.66.86.114: IP6 (hlim 252, next-header TCP (6)
payload length: 40) 2001:123:6d:727:222:4dff:fea4:2c0.39281 >
2001:123:5203:2b64:216:3eff:fe4c:91cb.18051: Flags , cksum 0xa99f
(correct), seq 3621908408, win 14120, options [mss 1412,sackOK,TS val
2303280641 ecr 0,nop,wscale 4], length 0
04:45:02.547579 IP (tos 0x0, ttl 255, id 12494, offset 0, flags [DF],
proto IPv6 (41), length 100)
    216.66.86.114 > 1.2.3.98: IP6 (hlim 253, next-header TCP (6)
payload length: 40) 2001:123:6d:727:222:4dff:fea4:2c0.39282 >
2001:123:5203:2b64:216:3eff:fe4c:91cb.18051: Flags , cksum 0xa871
(correct), seq 1353741932, win 14120, options [mss 1412,sackOK,TS val
2303281068 ecr 0,nop,wscale 4], length 0

We shouldn't be seeing traffic from 2001:123:6d:727::/64 from
92.50.66.98.  Now, if the anti-spoof code was turned on to its strictest
setting, that traffic would be completely dropped.

[...]

[mebbrecht]

My answer:

I think the QoS comes from our provider (UnityMedia) - there are no QoS
settings on this router. Sorry for that.

When do you see last traffic from 2001:123:6d:727::/64 on tunnel ID
XXXXXX - there is no date .... ? This is my home network. Perhaps the
traffic came over a VPN connection. I deactivated ipv6 on VPN
connections and there should be no more "wrong" traffic from now on (and
the last 10+ days....).

Is it possible, that my tunnels are banned ?! Could you unban
these tunnels, please ?

IMPORTANT: I'm quoting this mails, because I think it could be usefull for other users ... no to bash HE - there very helpfull for a free service ... good people :)

[PatrickDickey]

I'm not sure if this applies to your situation, but it might be useful for others also. Check in your account settings (on tunnelbroker.net) to make sure your IPv4 endpoint is correct. I had a case where my tunnels were supposedly updating, but I lost ipv6 access. I found out that they had changed my endpoint, so I was updating to the wrong server.

Have a great weekend.:)
Patrick.

[mebbrecht]

That wasn't the problem, but thanks.

Regarding to the support, the reason could be the following:

"Main thing that's unusual is the set TOS on the IPv4 traffic,
since that isn't usually set or processed past a network edge.  This
side complains due to there being TOS on the v4, but not on the v6 and
then gets hung up about that."

I disabled all yos stuff on my side - without any success. Later I got the following response:

"That said, I've seen this same thing from two other users, and there's
one thing you all have in common: your ISP.  All the traffic seems to be
getting marked as TOS class 3, which is defined as "Flash, most urgent.
Voice signaling".  Do they do combined VoIP and data services?  If so,
and you're using some routing of theirs upstream, it would seem like
their TOS marking routine is a bit... generous with what it tags.

Now that I have a common element, this should be much easier to track
down, and find what traffic from them is tickling the drops: whether
it's the TOS markings or something else."

It looks like my Provider:

UNITY MEDIA, Germany, NRW

Marks all traffic and that leads to this error. So if anyone got the same problem and ISP (UM Germany), refering to the ticket number 2471806 could be usefull when contacting the support.

If I get any updates on this issue, I'll post that here.

AND, MOSTLY IMPORTANT: GREAT SUPPORT, THANK YOU VERY MUCH, HE. (Thumbs UP!!!!)

[mebbrecht]

"I found a few other tunnels with this issue, and pushed out a rules
update on our side to just rewrite inbound tunnel traffic ToS to 0,
which seems to be resolving the issue."

Sweet!