Strange TTL behaviour

[kasperd]

I was doing some experiments with TTL settings and saw how some router was treating my packets. I thought I'd like to share this, and ask if anybody else have seen similar behaviour, and maybe knows which router may be having like this.

I have done a traceroute from my computer to the tunnel server and found out, that it is 12 hops away. If I set the TTL in the IPv4 header on the packets going towards the tunnel server to less than 12, I get an ICMP error from one of the routers I saw in the traceroute output, indicating that the packet couldn't make it to the tunnel server, as expected --- most of the time.

If however the TTL on the packet as it leaves my network is exactly 5, the encapsulated IPv6 packet does make it to the destination. The moment where I realized that using a TTL of 5 actually works even though the destination is 12 hops away was a real WTF moment. I also figured out how that happens.

At the moment all the 6in4 packets leaving my network has a TTL of 5 on the IPv4 encapsulation. They are not actually reaching the tunnel server, but the IPv6 packet inside does reach the destination. The router where the IPv4 TTL expires somehow treats an expired TTL on a 6in4 packet differently from an expired TTL on any other kind of packet. Instead of sending an ICMP error message, like you would normally do, that router simply unpacks the 6in4 pakcet and forwards the IPv6 packet from inside.

I verified this by running an IPv6 traceroute (while the IPv4 header has a TTL of 5 as it leaves my network), and confirmed that the IPv6 hops I see are totally different from what they normally look like. Instead of seeing Hurricane Electric IPs in the traceroute6 output, I see IPv6 addresses belonging to the same network as the IPv4 address that I would have expected to receive the ICMP error message from.

Does anybody here know what sort of router could be behaving like that?

[kasperd]

After much debugging I figured out that the router in question also decapsulates 6in4 packets, if the IPv4 header has an option field. That can cause really strange behaviour when trying to route packets between 6to4 addresses. Even though the packets I was trying to send between two networks should have taken a direct path, when I was using 6to4, adding an option field to the IPv4 header caused the following to happen.

Packet takes five hops along the proper path. This router notice an option field and decapsulates the 6in4 packet, even though the destination IPv4 address is not this router. It then forwards the packet on the IPv6 network. Since the destination address is a 6to4 address, it gets forwarded to the nearest public 6to4 relay (which happens to be in a different country).

I had heard handling of options in IPv4 headers was broken. I had no idea it was that bad. Maybe I should go and delete the code I wrote to generate an IPv4 option field, considering that it is unlikely to be any use to me now.

I guess the broken router can handle 6in4 packets just fine as long as it is handling them in hardware. But as soon as it has to handle packets in software due to TTL expiring or an option field present in the header, it gets processed by some buggy code, which decapsulates all 6in4 packets regardless of all other header fields such as destination IP. Or maybe it is actually a hardware flaw, and the route can decapsulate 6in4 packets in hardware. It might first try to forward the IPv4 packet in hardware, when that fails it may next try to decapsulate the 6in4 packet and handle the IPv6 packet in hardware, which then succeeds for my packets, but in reality it was supposed to be treating my packets in software instead.