MTU best practice?

[realdreams]

I set up the sit tunnel on RouterOS with the default /64 and advertise the a /64 from the /48 assignment on the LAN. I set MTU on the LAN to 1280. Local ipv6 traffic can have a higher MTU. Without LAN MTU 1280, I see ICMPv6 packet too large occasionally and it seems to cause issues with DNS zone transfer(payload length higher than 1280). I don't think it should cause connectivity issues since ICMPv6 is not being filtered, but it might cause delay and break some poor implementations?
What's the best MTU setting in this kind of scenario? Thanks.

[kasperd]

What's the best MTU setting in this kind of scenario?

Are you looking for a specific number to configure as your MTU? Or are you looking for advice on how to get your LAN traffic to use larger packets than your tunnelled traffic?

As for a good size to use, it is really a question of whether you want to have a value, that just works, or if you want to go through the hassle of figuring out the threshold at which problems appear. 1280 is the size, which is guaranteed to work according to the standard, and it is the lowest MTU you are allowed to configure on an interface. So if you want something that just works, then that is the MTU you go for.

If you want to do extra work to squeeze out the small performance boost you can get from going from 1280 closer to the 1480 that is the best you can hope for on the tunnel, you'll have to figure out what size you can really do on your tunnel and if you can do that size on paths, that matter to you.

As for using different MTU for LAN traffic and tunnelled traffic, the proper solution would be to use 1500 on the LAN and only use a smaller MTU on the tunnel interface, and then let PMTU discovery do the rest. But when PMTU doesn't work due to broken implementations, you have to do something else. A smaller MTU on the LAN can help for packets that you send. Packets that you receive could be worse off.

If the problematic traffic is TCP, then there is a solution, which is cheating a little bit, but works great. That solution is MSS clamping. On all TCP SYN packets going through the tunnel, your router modifies the MSS to ensure that both endpoints will know to not use packets larger than can fit in the tunnel. An advantage of this over lowering the MTU is that you are not introducing new problems by rejecting packets, which could otherwise have made it through.

If the problematic traffic is UDP, then solving the problem is trickier. AFAIK there is no way around retransmitting the UDP packet at the application level, when an ICMPv6 too big message is encountered. I am not sure if the application layer is even notified about this, or if it will have to time out and retry.

Unless you use certain extensions, DNS has a packet size limit for UDP way below the 1280 byte minimum MTU required by IPv6. As such you can avoid those problems by not using DNS extensions for larger packets, or configure it such that you don't permit more than 1232 bytes of DNS payload.

But AFAIR DNS zone transfers are supposed to use TCP without even trying UDP, so maybe something else is going on.

[kriteknetworks]

Unless he means EDNS for dnssec lookups, and not transfers, thats UDP....

[realdreams]

Unless he means EDNS for dnssec lookups, and not transfers, thats UDP....

The problem I had in the past was zone transfer to dns.he.net.
I am running master dns with Windows Server 2008 R2 and dns.he.net as slaves. The zone never gets updated without setting MTU=1280 on the router LAN interface (where the master DNS is). Wireshark on the master server shows that for every axfr attempt after tcp handshake and I get a ICMPv6 packet too large(payload was like 1292) for the axfr response, MTU recommended 1280. But the packet resent is exactly the same as the original, but this time I don't see ICMPv6 Packet Too Big. Unless Wireshark is doing something wrong, I don't get why PMTUD didn't change MTU and why I didn't see ICMPv6 Packet Too Big this time... I remember Wiresharking complanting something about packet size exceeding capture limit tho.
That's what gets me confused.

modifying MSS can be done with ip6tables? Does it have a big performance impact on the router (say like 20000 concurrent connections)?

[kasperd]

Wireshark on the master server shows that for every axfr attempt after tcp handshake and I get a ICMPv6 packet too large(payload was like 1292) for the axfr response, MTU recommended 1280.

I don't get why the recommended MTU is so small. You should be able to do larger than that, as long as it stays inside the HE network. Does the ICMPv6 packet come from your own router?

It is not really important where the ICMPv6 packet came from though or what the recommended size is. As long as the ICMPv6 packet gets through, things should just work.

But the packet resent is exactly the same as the original

That sounds like a bug in Windows.

but this time I don't see ICMPv6 Packet Too Big.

That could be because the router is rate limiting them.

Unless Wireshark is doing something wrong, I don't get why PMTUD didn't change MTU and why I didn't see ICMPv6 Packet Too Big this time...

This is definitely something the Linux version of Wireshark could handle just fine. But I seem to recall there being certain limitations to packet captures in Windows.

One possible explanation is that Windows is handing the full packet to Wireshark before it does fragmentation. That is just a guess though, I have no idea if Windows would do that. I suggest you try dumping the packets at the receiving end of that connection. For example you can use the tcpdump command line utility to dump packets into a file. That file can then be loaded into Wireshark for analysis. If you run tcpdump on Linux and display it using Wireshark on Windows, you shouldn't be affected by any of the packet capture limitations of Windows.

I remember Wiresharking complanting something about packet size exceeding capture limit tho.

I have never seen that complaint from Wireshark.

modifying MSS can be done with ip6tables?

According to documentation it can be done.

Does it have a big performance impact on the router (say like 20000 concurrent connections)?

Modifying MSS shouldn't necessitate connection tracking. I have however seen cases where iptables would not do certain operations without connection tracking, even though they should work perfectly well without connection tracking.

I don't know if ip6tables can do MSS modifications without connection tracking. If it means tracking 20000 concurrent connections, I would be a bit worried, and look into other ways of modifying the MSS.

As long as you avoid connection tracking, the number of connections isn't important. Then the question is how many packets per second the router can handle. Since you mention iptables, I assume this is a software router. Modifying the MSS does increase the CPU usage per packet a little bit, since it must inspect every packet to figure out if it is a TCP SYN packet. Additionally lowering the MSS means smaller packets, which means more packets needed to transfer the same amount of data. Whether the increased CPU usage due to that is a problem, is something you'd have to benchmark yourself.

If your WAN connection is 100Mbit/s or less, I wouldn't worry. If your WAN connection is 1Gbit/s or more, I'd expect a software based router to be limited by CPU resources.