No working MTU can be found! Was: MTU issue?

[vsauer]

Hi folks,

I have a strange problem with one of my IPv6 uplinks.
The uplink is a VDSL100 of Deutsche Telekom with PPPoE and MTU of 1492 terminated on a linux-based PC (Brand "PC Engines" model APU2 with Ubuntu 14.04).

Here's the uplink interface:

ppp0      Link encap:Point-to-Point Protocol
          inet addr:84.119.111.111  P-t-P:62.155.246.13  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1

Basically, I set the MTU of the he-ipv6 interface as well as in the tunnelbroker web-interface to 1472.
The tunnel goes to Frankfurt (216.66.80.30).

he-ipv6   Link encap:IPv6-in-IPv4
          inet6 addr: fe80::5495:b377/64 Scope:Link
          inet6 addr: 2001:470:1111:aaaa::2/64 Scope:Global
          UP POINTOPOINT RUNNING NOARP  MTU:1472  Metric:1


root@bh:~$ ip tun sh
he-ipv6: ipv6/ip  remote 216.66.80.30  local any  ttl 255  6rd-prefix 2002::/16

In iptables and ip6tables I enabled mss-clamping for ppp0 or rather he-ipv6.

Ping works but TCP-Connections get interrupted after a few packets which looks like an MTU issue.
Tracepath claims the MTU of 1472 as okay:

root@bh:~$ tracepath6 xxx
1?: [LOCALHOST]                        0.139ms pmtu 1472
1: xxx-2.tunnel.tserv6.fra1.ipv6.he.net              12.648ms
1:  xxx-2.tunnel.tserv6.fra1.ipv6.he.net              13.633ms
2:  ve399.core1.fra1.he.net                              16.649ms
[....]
7:  xxx2.kettenbach-it.de                                  12.957ms reached
     Resume: pmtu 1472 hops 7 back 7

Here's what ping with different packet sizes says:


root@bh:~$ ping6 -s 1472 -M do ipv6.google.com
PING ipv6.google.com(fra16s18-in-x0e.1e100.net) 1472 data bytes
ping: local error: Message too long, mtu=1472

root@bh:~$ ping6 -s 1425 -M do ipv6.google.com
PING ipv6.google.com(fra16s18-in-x0e.1e100.net) 1425 data bytes
ping: local error: Message too long, mtu=1472

root@bh:~$ ping6 -s 1424 -M do ipv6.google.com
PING ipv6.google.com(fra16s18-in-x0e.1e100.net) 1424 data bytes
72 bytes from fra16s18-in-x0e.1e100.net: icmp_seq=1 ttl=58 (truncated)


Obviously, packets larger than 1424 do not go through and packets smaller or equal 1424 get truncated.

If I lower the MTU to the minimun of 1280, the pings look like this:


PING ipv6.google.com(fra16s18-in-x0e.1e100.net) 1280 data bytes
ping: local error: Message too long, mtu=1280

PING ipv6.google.com(fra16s18-in-x0e.1e100.net) 1233 data bytes
ping: local error: Message too long, mtu=1280

PING ipv6.google.com(fra16s18-in-x0e.1e100.net) 1232 data bytes
72 bytes from fra16s18-in-x0e.1e100.net: icmp_seq=1 ttl=58 (truncated)



Wha do packets always get truncated???

I don't know what's going on here and I don't know could be the next (debugging) steps.
Does anybody have a hint for me?

Volker

[divad27182]

Truncation?  Well, I think it is because google's box is truncating the reply to 64 bytes of data.
Try pinging ipv6.he.net instead.

Truncating the reply is a reasonable sanity measure.  It allows you do send a 64 byte reply to a 64000 byte request, thus no clogging up your own outgoing packets.  It also makes it more likely for the reply to get through an asymetric MTU problem.  On the other hand, one gets problems like what you are seeing.

My issue with all this is that wireshark doesn't manage to pair up the truncated reply with the request.  Somebody never saw a truncated reply.

edit: That 48 byte discrepancy is the IPv6 header (40 bytes) and the ICMPv6 header (8 bytes).  The byte count specified is the payload length.

[vsauer]

Ah, okay!


root@bh:~$ ping6 -s 1424 -M do ipv6.he.net
PING ipv6.he.net(ipv6.he.net) 1424 data bytes
1432 bytes from ipv6.he.net: icmp_seq=1 ttl=58 time=152 ms

root@bh:~$ ping6 -s 1425 -M do ipv6.he.net
PING ipv6.he.net(ipv6.he.net) 1425 data bytes
ping: local error: Message too long, mtu=1472


1424 is the maximum size of a packet that can go through.
MTU of the link is 1472.

If I do an ssh from one tunnel endpoint on the frankfurt server (with 1480 bytes, because it's nit PPPoE) to another having PPPoE, the ssh session gets stuck after some bytes.
ssh from the 1480 MTU to a internet server with native ipv6 works fine.
ssh from the 1472 MTU gives the same problem  - ssh gets stuck.

What's wrong with the 1472 MTU uplink. How can I debug it?

[divad27182]

1424 is the maximum size of a packet that can go through.
MTU of the link is 1472.

Close, but not quite.  1424 is the maximum payload length that, with the addition of 48 bytes of header, can fit in an MTU of 1472.  1424+48=1472  Specifying 1424 actually gets you a packet length of 1472.

What's wrong with the 1472 MTU uplink. How can I debug it?

Myself, I would probably start by "tcpdump"ing to a file on both end, and then analyzing the files with wireshark.  Look for what differs.  Look of ICMP messages.

Code Select Expand

# tcpdump -i eth0 -s 9999 -w first_end.tcpdump tcp port 22 and host otherhostname

--David

[vsauer]

I did some debugging with wireshark:

For MTUs > 1280, I get a ICMP6 message "packet to big".
Here's a test with MTU 1284:

5   0.156892   2001:470:1f0a:127b::2   2001:470:1:18::1281   HTTP   236   GET /ip/?size=1600&fill=xx HTTP/1.1
6   0.307349   2001:470:1:18::1281   2001:470:1f0a:127b::2   ICMPv6   1280   Packet Too Big


For MTU==1280, which seems to be the MTU suggested by ICMPv6, my connection get's reset after a some time, even the payload was delivered correctly.
See screenshot.



ssh session and most other TCP-sessions don't work this way.

What is wrong here?

[vsauer]

Maybe the same as this:
https://forums.he.net/index.php?topic=3601.0

which links to this:
https://ttlexpired.co.uk/2016/02/12/ipv6-tunnel-and-failing-tcp-sessions/

So it may be a problem close the the VDSL2 hardware.
The problem description fit's exactly, except hat tcpdump -p doesn't help in my case. But this makes sense, since my router and the modem are different devices.

Right now I'm using a AVM FritzBox as VDSL2 Modem/Bridge. It has a Broadcom Chip. FastPath etc. can not be configured, since this is a consumer router.

Before, I used an Allnet VDSL2 Modem. IPv6 was working stable. It doesn't use a Broadcom but a Lantiq Chipset. Unfortunately, this thing was not running stable on the VDSL2 layer as other products based on this chipset as well so I had to return it.

I now ordered one of these:
http://www.zyxel.com/de/de/products_services/vmg1312_b_series.shtml?t=p&tabOrder=2

This is the product Deutsche Telekom uses at as business customer CPE.
This might fix the problem and be a much better solution than the FritzBox anyway.

[divad27182]

I did some debugging with wireshark:

What is wrong here?

Well, several packets from the remote server were lost.  The remote webserver tried to end the connection normally after 5 seconds of inactivity.  A TCP retransmission was initiated, and some (but not all) data was successfully resent.  Then (assuming TCP's rules haven't changed since IPv4) either the remote machine was rebooted, or it's TCP stack is broken or at least poorly configured, as those are the only way you get a RST in 120 seconds.  [I wouldn't bother with this issue.] 

Both ends seem to be sending a payload length (inside all the headers) of 1208, with an overall packet length of 1280, and you did receive such a packet.

Well, TCP did always require only a non-zero probability of successful delivery from it's lower level, but it never really behaves well unless it gets a very high probability of successful delivery.  Maybe you have a noisy connection (which seems unlikely with current technology).  Most of the dropped packets appear to be incoming.  If so, you might be able to tune your TCP stack to be more aggressive at sending and requesting retransmissions.  Unfortunately, that is just a workaround.

As for the underling problem?  Your guess is as good as mine, but I would suspect it isn't MTU, as the payload 1208 overall 1280 packets went both ways successfully (#4, #10).

[vsauer]

Hallo all,

indeed, replacing the FritzBox 7430 by the Zyxel VMG1312-B30A as Modem-Bridge solves the problem.
The (in germany) very popular FritzBox seems to mess up TCP-Sessions in IPv6, as described in https://ttlexpired.co.uk/2016/02/12/ipv6-tunnel-and-failing-tcp-sessions/ (here by some other kind of CPE).
It has nothing to do with MTU.

So if you want a static prefix for your ipv6 (which Deutsche Telekom won't give you) over a linux-router using a modem-bridge for vdsl(2), you'd better get something else then a FritzBox.
Don't get the Allnet BM100VDSL2 with Lantiq chipset, either. My specimen did not run stable on the VDSL layer and it's software is really a piece of crap.

The Zyxel VMG1312 does a good job. In fact the best of the 4 products it testet.

Hope that helps everyone having these kind of problems!

Volker



And here's an explantion in german, since the FitzBox is very popular in Germany and this needs to said in german, too, so Google will find it:

Das Problem ist:
eine FritzBox 7430 wird als Modem+Bridge an einem VDSL2-100 Anschluss der Deutschen Telekom AG (DTAG) eingesetzt.
Das wird zwar angeblich offiziell nicht mehr unterstützt, geht aber (Juni 2017) trotzdem, wenn man in den Interneteinstellung als Provider "Sonstige" verwendet, angibt das keine Authentifikation notwendig ist und dann PPPoE-Passthrough einschaltet.
Die FritzBox entfernt dann das VLAN7 der DTAG und stellt PPPoE Pakete auf den LAN-Ports zur Verfügung.

Auf den angeschlossen Router läuft Ubuntu 14.04 mit dem ppp-daemon.
IPv4 läuft einwandfrei stabil.
Legt man über tunnelbroker.net einen IPv6 Tunnel an, so funktioniert Ping sowie einige UDP-Protokolle.
TCP-Session brechen jedoch nach wenigen Paketen ab.
Siehe dazu ältere Beiträge dieses Threads.
An der MTU liegt es nicht.
Der Hinweis auf das eigentliche Problem findet sich hier: https://ttlexpired.co.uk/2016/02/12/ipv6-tunnel-and-failing-tcp-sessions/
Scheinbar macht der Chipsatz der FritzBox 7430 in dieser speziellen Situation Probleme.
Da ein Zugriff auf den Chipsatz - im Gegensatz zu der CPE in dem verlinkten Artikel - nicht möglich ist, blieb nur das Austausch der Box gegen ein Zyxel VMG1312-B30A. Dieses wird von der DTAG an Businesskunden, die eigene  Router ohne Moden einsetzen, veermarktet.
Das Zyxel gerät wird ebenfalls nur als Modem-Bridge mit VLAN-stripping verwendet.
Mit diesem Gerät funktioniert dann alles: VDSL2, IPv4 und IPv6 über he.net einwandfrei.