problems with packet size limits through my HE tunnel?

[IMELT]

I have an ASUS RT AC66U connected to a Huawei fiber optic router that is in bridge mode for the Internet part of my connection. The ASUS has a very easy 6in4 setup, which I have done to connect to my tunnelbroker tunnel. And yet I can't connect from any of my computers / devices to the internet using IPV6.

Here is what I have done and discovered that seems relevent:

-- When I set up the ipv6 settings and apply them on the router, I can ping through ipv6.  That is, ping -c 3 ipv6.google.com works fine when telneted to my router and also directly from my computer. So the computer and the router are talking, and the tunnel is up.  I can also traceroute the same site with no problem. I can also force ipv6 dns checks successfully, i.e. not get the ipv6 address from ip4 servers. So DNS over ipv6 can get through the tunnel.

-- As soon as I try to do anything else that might use ipv6, there is a long delay, the computer (browser, whatever) eventually reports some failures (say google can't establish it's secure connection), then reloading it reverts to using ipv4.  IPV6 testing sites report only failure, and most interestingly to me, the tunnel seems to have broken. That is, after trying to use ipv6 over the web, I can NO LONGER ping from the computer or from the router through the 6in4 tunnel.  Resetting the tunnel by changing anything (say ttl down or up by 1) and applying will reset the tunnel, ping will work again, always until anything else tries to use the tunnel.

-- My ISP uses PPPOE, and requires vlan6 tags (because the connection is really not only for internet, and that, oddly, is the VLAN tag number they have chosen for internet. That connection has an mtu and mru of 1492 and works fantastically for everything except what we are talking about.

-- Noticing on the advanced page of the tunnel details that the MTU sometimes needs to be reduced for some pppoe connections, I tried doing so.  It didn't fix the problem.  So I used the -s flag in ping and started testing different packet sizes to see if the pings broke at some particular threshold. THEY DID.

-- If I do nothing else, ping works up to a setting of 1230 bytes, where 1238 bytes are being returned in the icmp responses. Anything higher and I get 100% packet loss. That is, ping -s 1230 works, but ping -s 1240 will time out and announce 100% packet loss. Doing this once with three packets sent doesn't break the tunnel, I can still then ping successfully at 1230 bytes again, but I am nonetheless guessing that if it were flooded with failures, as would happen if I tried to actually do anything useful over ipv6, that would explain why the tunnel was being closed, presumably at HE's end.

-- The minimum setting for MTU is 1280. Not just on HE's settings. In my router, as well, and in general. I am new to ipv6, but it is also my understanding that ipv6 will probe MTUs for an entire route and then send. Packets are not fragmented along a route as they can be in ipv4.  So if any stage of my connection can't receive packets that others will build, nothing will ever be sent to me.

-- Maybe I'm barking up the wrong tree.  Maybe the ping behavior has nothing to do with my problem. I'm flailing here.

Any answers, ideas, or hints at directions to explore will be GREATLY appreciated.

[kasperd]

That connection has an mtu and mru of 1492 and works fantastically for everything except what we are talking about.

ping works up to a setting of 1230 bytes, where 1238 bytes are being returned in the icmp responses. Anything higher and I get 100% packet loss.

These two pieces of information seem to be conflicting. The relevant headers, which could explain a difference between the numbers are: 20 bytes IPv4, 40 bytes IPv6, 8 bytes ICMPv6. But that only add up to 68 bytes, which is not nearly enough to explain the difference from 1230 to 1492, which is as much as 262.

That is not to say that your analysis is incorrect. It just means there is an explanation somewhere, which need to be found. It may be that you do have 1492 bytes of MTU on the first hop, but at a later hop in the ISP network, the MTU is lower. If all your communication is affected by this and other traffic is unaffected, it means PMTU discovery need to be working for your IPv4 communication with all sites. It sounds a bit unlikely, that you'd never run into an IPv4 site which breaks PMTU discovery one way or another. If the ISP does MSS clamping on your IPv4 traffic, they can effectively hide an PMTU problems. This is a lot of guessing, it is unlikely that I got it spot on.

I am nonetheless guessing that if it were flooded with failures, as would happen if I tried to actually do anything useful over ipv6, that would explain why the tunnel was being closed, presumably at HE's end.

Connections are stateless and as such are not being closed. Any proper 6in4 implementation does implement PMTU discovery for the IPv4 path either directly on the 6in4 layer or by pushing errors to the IPv6 layer. It sounds plausible that too many errors is causing something to give up. If changing a setting on your own router makes the tunnel work again, that must mean it was your own router, which had given up.

The minimum setting for MTU is 1280. Not just on HE's settings. In my router, as well, and in general.

Correct. That is specified in the IPv6 standard. If we add the size of an IPv4 header to this, we get that you need an IPv4 MTU of at least 1300 bytes in order to run a tunnel without problems. With the dominant position of Ethernet you usually have an MTU of 1500 bytes or at least close to it. So the 1300 bytes requirement is rarely a problem.

Since it sounds like your IPv4 MTU may actually be less than 1300 bytes, it is going to be tricky for you to get it working. Fragmentation on the IPv4 layer of 6in4 packets is permitted, but such fragmentation is not supported by all 6in4 implementations, and if anycast addresses are in use on either end of the tunnel, it is unlikely to work. Besides it is not desirable to use fragmentation on the IPv4 layer, even if you could avoid those problems.

For TCP traffic there is a better solution, which is to reduce the MSS. Even though IPv6 requires the MTU to be at least 1280 bytes, it does not make any similar requirements to MSS. If you can get your router to reduce the MSS on all TCP SYN packets going through the router, you can probably avoid most of the problems you are seeing. I'd first try setting MSS to 1024 to see how it works out.

[IMELT]

Yikes! Thanks for the fast and extremely useful answer. In my router I used iptables to set the mss on tcp syn packets to 1024 as you suggested, and I have full access to everything ipv6 and flying colors on all the tests. Now if only ISPs in Spain really start supporting ipv6 natively so we don't have to resort to this sort of arcana (which, of course, slows things down), I'll be even happier. But ...

thanks VERY much.

[IMELT]

HUGE AND IMPORTANT EDIT!!! The whole question was wrong. Immediately after posting the question I had an epiphany. ICMP is notoriously problematic for its DOS potential etc., ... And it suddenly occurred to me that maybe some hosts reject large ICMP ping requests. So I tried pinging other servers with large size requests. And some worked.

Hah. So, I learned a lot from your posts and my subsequent reading, but all, apparently, unnecessarily.  The pings weren't coming back because some servers refuse to send echo responses above that size, nothing else.

The only remaining question was why ping was working when other traffic was not .. for a while. But for that I stick with what I said below about some intermediate problem that cleared up either on restore or by the route/setting timing out of existence.

So, sorry to have wasted your time, and thanks to leading me to a mini-education in a whole set of issues. I'll stop talking to myself in the forum now.


**** ----- pre-edit from here ----- ***


Hmm. Well. I get the feeling I jumped the gun with my questions and wasted your time, though I still don't understand what was happening. And I'm still way too inexpert to understand the results.

And yet here I am inviting you to waste a moment longer on me if you feel the inclination.

After your post I did as I said, setting a low MSS and everything worked. Great. But I wonder if it was actually a coincidence.

I still didn't understand some things, so I went and read up a bit, e.g. about MTU, MSS, PMTU and its relation to ICMP and don't fragment flags in headers, and RFC 4213 on ipv6 transition mechanisms including 6in4, and another on ip in ip encapsulation tunnels (although 6in4 has at least one MUST that is contrary to the ip over ip encapsulation rfc, which makes sense: flag to allow fragmentation even when the ipv6 header doesn't ... but then specifies only to do the fragmentation at the ipv4 tunnel level when the pmtu is too small to request smaller packets at the ipv6 level, etc., ...

That is to say, I read up, within my capacity to understand these things, given that it isn't an area of expertise for me.

Long story short, my experience still didn't make sense to me. For example, why was I not ever having problems in ipv4 UDP? My ISP could, as you say, fix a low MSS for TCP, but for UDP, if PMTU was so small and not being correctly reported, why wasn't I losing a lot of packets or seeing issues with communications that use UDP? Anyhow, wouldn't it be highly unusual for the PMTU to be THAT low? Even taking into account that the PPPOE has embedded in it, if I understand correctly, the extra 30 bytes from using the modified ethernet packets that allow vlan and priority tagging, it still doesn't come close to adding up.

This morning I reset everything on my router, did NOT set the MSS, just left it clamped to the PMTU, redid all the settings (exactly the same, confirmed with screenshots, lol), and IPV6 STILL WORKS!! Without setting the MSS low.

My first thought was, ok, this has to do perhaps with my newish account with HE, or more likely with some strange relic of some intermediate incorrect setting or route or something when I was initially configuring my router, something somewhere that cleared out either with the restore or by timing out, but great, now it works.

But it leaves me with one big remaining question that I am hoping you can answer for me, although I understand that you probably would rather spend your time resolving a real problem that someone has with something broken.

Why is it still true that if I ping an ipv6 site with a packet size of more than 1230 I don't get a response? Even if I am right about the extra 30 bytes from vlan/qos tagging, which I am not entirely clear on, I am only requesting a packet of 1230 + 68 + 30 = 1328 bytes over a pppoe connection that, in principle, has an mtu of 1490. It all doesn't add up to the result I am getting.

I should probably just chalk it up to isp/router manuf./mixed network esoterica, and then wipe the question from my brain. But have a deep compulsion to get to the bottom of things that often bullies me into ignoring how much practical knowledge is missing between my question and the accessibility of the real answer.

So, do you have any idea what is going on?

[kasperd]

ICMP is notoriously problematic for its DOS potential etc.

Descriptions of that class of problems are usually exaggerated. Overzealous filtering of ICMP tend to cause more problems than ICMP did in the first place. Rate limiting of ICMP replies is sometimes a good idea. In particular routers that can forward packets in hardware but need to send ICMP replies in software will have to rate limit such replies. That is mostly visible when using traceroute and similar tools.

And it suddenly occurred to me that maybe some hosts reject large ICMP ping requests.

Some hosts do reject large echo requests. AFAIR Google rejects any echo request larger than the 64 bytes, which the ping command sends by default. Some hosts will accept echo requests op to the minimum required MTU of 1280 bytes, and yet other hosts will respond to echo requests as long as they are not fragmented.

For example, why was I not ever having problems in ipv4 UDP?

UDP packets tend to be smaller. Though the standard permits UDP packets up to 64KB in size, hosts are not required to implement support for that large UDP packets. In fact with IPv4 hosts are only required to implement support for up to 576 bytes (leaving room for 548 bytes of payload). Protocols that need to send more than 576 bytes at a time are typically designed to run on top of TCP rather than UDP. Moreover on IPv4 UDP packets typically permit fragmentation in-flight. Routers that would silently drop too large IPv4 packets with DF set may still correctly fragment them if DF is not set.

Anyhow, wouldn't it be highly unusual for the PMTU to be THAT low? Even taking into account that the PPPOE has embedded in it, if I understand correctly, the extra 30 bytes from using the modified ethernet packets that allow vlan and priority tagging, it still doesn't come close to adding up.

You are absolutely right. It would be highly unusual for the PMTU to be that low. Problems are rarely caused by an MTU being too low for a 1280 byte IPv6 packet even with encapsulation, more often it is caused by sending packets which are just a few bytes too large and not getting notification about that fact.

1280 is large enough that limiting the MTU on IPv6 links to 1280 is not much of a problem. And by going as low as 1280 bytes, you know that there won't be a lower MTU elsewhere on the path, and you get better control over behaviour, i.e. you can ensure ICMP errors are send by yourself rather than forwarding the packet to somebody else, who may silently drop it. For that reason it is possible to see networks, that use a 1280 byte MTU for IPv6 but a higher MTU for IPv4. The 1280 byte threshold does not apply to IPv4, so on IPv4 you would not be guaranteed to be the lowest MTU by setting it to 1280. The equivalent number for IPv4 is only 68 bytes, which is so ridiculously low that you don't lower your MTU from 1500 bytes to 68 bytes to avoid potential problems.