Netalyzer says I have a IPv6 fragmentation problem.

[bicknell]

I installed a new home router today (an Apple Time Capsule, if you care) and as part of that work ran some tests on my network connectivity.  One of the tools I used was the ICSI Netalyzer, available at http://netalyzr.icsi.berkeley.edu/.

It detected an IPv6 MTU issue:

  IPv6 Path MTU (?): Warning

  Your system can not send or receive fragmented traffic over IPv6. The 
  path between our system and your network has an MTU of 1480 bytes. The
  bottleneck is at IP address 2001:470:0:90::2. The path between our   
  system and your network does not appear to handle fragmented IPv6     
  traffic properly.

They also have a link with more information on thi particular error, it is http://n2.netalyzr.icsi.berkeley.edu/info_ipv6_mtu.html.

Now the IPv6 address is the native interface of the HE Tunnel server my tunnel terminates on in Ashburn.

I think there are two possibilities:

1) The Tunnel MTU is smaller than 1500, so the HE Tunnel server should return an ICMP Packet Too Big, but perhaps those are not being generated, being filtered, or being rate limited and too many are showing up during my test.

2) Fragments are correctly sent down the tunnel, but the Time Capsule is dropping them due to some Apple bug.

I tried to figure out a fast way to test, but couldn't come up with anything super-easy to see if this was a known issue (with either HE or the Apple Airport/Time Capsules).  Anyone?

[kasperd]

The problem doesn't have to be at one of the tunnel endpoints. It could be on the path between them. Also, it is not clear in which direction there is a problem. It may be that too large packets are dropped in only one direction, and in the other direction every packet will either make it through or result in an ICMPv6 packet being returned. Since you haven't mentioned what your IP address is, I cannot investigate the problem myself.

You should try to ping the tunnel server with various packet sizes and run tcpdump to figure out what is going on. Do this with both IPv4 and IPv6. Find out what the path MTU appears to be in each case, and what happens when it is exceeded.

[cholzhauer]

I ran it too just to see what results I got.  Along the same lines as you:

Code Select Expand


Your system can not send or receive fragmented traffic over IPv6.
The path between your network and our system supports an MTU of at least 1280 bytes. The path between our system and your network has an MTU of 1480 bytes. The bottleneck is at IP address 2001:470:0:6e::2. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.


[kasperd]

I got the test running as well. This is what I got:

Your system can not send or receive fragmented traffic over IPv6.
The path between your network and our system supports an MTU of at least 1472 bytes. The path between our system and your network has an MTU of 1480 bytes. The bottleneck is at IP address 2001:470:0:69::2. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.

I tried to disable the HE tunnel and run with just 6to4. Then I got this:

Your system can not send or receive fragmented traffic over IPv6.
The path between your network and our system supports an MTU of at least 1472 bytes. The path between our system and your network has an MTU of 1450 bytes. The bottleneck is at IP address 2001:1900:5:1::229. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.

And one really strange thing happened when I was running just 6to4, I got this message:

Your host, NAT, or firewall acts as a DNS server or proxy. Requests sent to this server are eventually processed by 216.66.80.90.
This is probably a bug in your NAT's firmware, and represents a minor security vulnerability.

That last message was without using the HE tunnel, and with resolv.conf listing only my ISPs DNS servers. I have no idea where it got that IP from. The test generates too much traffic for me to be bothered with going through it all. I don't experience any MTU related problems, so I am not sure what it is this test is picking up.

Maybe if there is a test site somewhere that has native IPv6 and can ping me with a packet size that I choose, I could figure out a bit more.

[bicknell]

Are both of you running Apple routers (Apple Extreme Base Stations or Time Capsules)?

I'd like to get a couple of people with non-apple products to try, as if it occurs with them it would point more towards HE being the problem, and if it is Apple products only it would point to Apple's implementation.

[cholzhauer]

Nope, FreeBSD 8.2 router

[broquea]

I remember getting similar results while on NATIVE IPv6 in the HE NOC. Running it from my Ubuntu 11.10 laptop at home via tunnel terminating on dir-825 I get:

Code Select Expand

IPv6 Path MTU (?): Warning
Your system can not send or receive fragmented traffic over IPv6.
The path between your network and our system supports an MTU of at least 1480 bytes. The path between our system and your network has an MTU of 1480 bytes. The bottleneck is at IP address 2001:470:20::2. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.

Are they saying 1480 is bad? Because I'd love to know a tunnel that gives you 1500. Also I love that the "bottleneck" is at an anycasted IP. Actually thinking about it, I probably forgot to set preferred_lft 0 on the anycast ip on tserv29. Maybe whomever took over will double-check it.

Code Select Expand

109.551 test-42| Message: mtu 1488 64
109.552 test-42| UDP socket at 2001:470:67:22f:224:2cff:feaa:6383:46451
109.732 test-42| Got datagram of 29 bytes.
109.732 test-42| Responsive failure
109.732 test-42| Response is bad 1488 2001:470:20::2 1480
109.732 test-42| Works: 1476
109.732 test-42| Fails: 1488
109.732 test-42| At:    1482
109.732 test-42| Message: mtu 1482 64
109.733 test-42| UDP socket at 2001:470:67:22f:224:2cff:feaa:6383:45177
109.903 test-42| Got datagram of 29 bytes.
109.903 test-42| Responsive failure
109.904 test-42| Response is bad 1482 2001:470:20::2 1480
109.904 test-42| Works: 1476
109.904 test-42| Fails: 1482
109.904 test-42| At:    1479
109.904 test-42| Message: mtu 1479 64
109.905 test-42| UDP socket at 2001:470:67:22f:224:2cff:feaa:6383:56287
110.004 test-42| Got datagram of 1024 bytes.
110.005 test-42| Success
110.005 test-42| Works: 1479
110.005 test-42| Fails: 1482
110.005 test-42| At:    1480
110.005 test-42| Message: mtu 1480 64
110.005 test-42| UDP socket at 2001:470:67:22f:224:2cff:feaa:6383:35993
110.103 test-42| Got datagram of 1024 bytes.
110.103 test-42| Success
110.103 test-42| Works: 1480
110.103 test-42| Fails: 1482
110.103 test-42| At:    1481
110.103 test-42| Message: mtu 1481 64
110.104 test-42| UDP socket at 2001:470:67:22f:224:2cff:feaa:6383:36066
110.276 test-42| Got datagram of 29 bytes.
110.277 test-42| Responsive failure
110.277 test-42| Response is bad 1481 2001:470:20::2 1480
110.277 test-42| Final MTU is 1480

So yeah, its "bad" because it's limited to 1480?

[cholzhauer]

I just tried from home with my tunnel hosted on a D-Link DIR615

Code Select Expand


Your system can not send or receive fragmented traffic over IPv6.
The path between your network and our system supports an MTU of at least 1480 bytes. The path between our system and your network has an MTU of 1480 bytes. The bottleneck is at IP address 2001:470:0:5d::2. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.


[bicknell]

So yeah, its "bad" because it's limited to 1480?

Just to make my example more clear, assume a tunnel can move a 1480 byte packet max, and the "Internet" is 1500 byte clean.  I realize those are not always true, but I want a simple example to get us all on the same page.

Source system sends a 1490 byte IPv6 packet.  It should make it to the tunnel end point (e.g. one of HE's tunnel broker servers) which realizes the 1490 byte packet can't fit down the 1480 byte tunnel.  That router should send back an IPv6 ICMP "Packet Too Big" message to the source and drop the packet.  Note this is different from IPv4, an IPv4 router would split the packet in half and forward it, in IPv6 that does not happen.  Source system gets the ICMP "Packet Too Big" and is thus supposed to split the original 1490 byte packet into two parts and send a packet plus a fragment on to the destination.  These both now fit, and make it down the tunnel where the receiver reassembles the packet.

Your log now makes me think I've run into this problem before.  With TCP this all works nicely on all operating systems, because a TCP sender has to buffer the data until an acknowledgement is received.  With UDP, well, that's more interesting.  I believe there are operating systems that fire off the UDP packet and don't buffer the packet at all (as this was the IPv4 behavior), and thus when the Packet-Too-Big comes back there is no data to retransmit.  Also, this packet too big should create a PMTU entry (on operating systems with path MTU tracking) allowing subsequent packets to simply be fragmented at the source.

I would strongly hope the Netalyzer folks are using servers that do everything right, I mean, I assume their suite of tests 100% passes when they run it locally.  I think what we need to do is capture a TCPdump from both ends of a tunnel while running the test, and then look at the packets.

[bicknell]

I'm going to post a bit more info here:

Client side of my failed test:

163.903    main|
163.903    main| Running test 42: checkMTUV6
163.903    main| ----------------------------
163.913 test-42| Testing the ability to send a large UDP packet (2000 bytes) over IPv6
163.913 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.913 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53348
163.914 test-42| Got exception java.net.SocketException: Message too long on UDP test
163.914 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.914 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53349
163.966 test-42| Got exception java.net.PortUnreachableException: ICMP Port Unreachable on UDP test
163.966 test-42| Can't send UDP fragments
163.966 test-42| Testing the ability to receive a large UDP packet (2000 bytes) over IPv6
163.966 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.966 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53350
164.023 test-42| Got exception java.net.PortUnreachableException: ICMP Port Unreachable on UDP test
164.023 test-42| Can't receive UDP fragments
164.023 test-42| Attempting to send a packet with
164.023 test-42| fragmentation of 2009 bytes
164.023 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
165.024 test-42| No data received.
165.024 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
166.025 test-42| No data received.
166.025 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
167.025 test-42| No data received.
167.025 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
168.026 test-42| No data received.
168.027 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
169.028 test-42| No data received.
169.028 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
170.028 test-42| No data received.
170.029 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
171.029 test-42| No data received.
171.029 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
172.030 test-42| No data received.
172.030 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
173.032 test-42| No data received.
173.032 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
174.033 test-42| No data received.
174.033 test-42| No reply back
174.033 test-42| Now looking for the receive MTU. Trying 1500 first
174.033 test-42| MSG: mtu 1500 64
174.033 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53352
174.095 test-42| Got datagram of 31 bytes.
174.095 test-42| Response is bad 1500 2001:470:0:90::2 1480
174.095 test-42| Path MTU is <1500B
174.095 test-42| Beginning binary search to find the path MTU
174.095 test-42| Works: 0
174.095 test-42| Fails: 1500
174.095 test-42| At:    750
174.095 test-42| Message: mtu 750 64
174.095 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53353
174.148 test-42| Got datagram of 702 bytes.
174.148 test-42| Success
174.148 test-42| Works: 750
174.148 test-42| Fails: 1500
174.148 test-42| At:    1125
174.148 test-42| Message: mtu 1125 64
174.148 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53354
174.203 test-42| Got datagram of 1024 bytes.
174.203 test-42| Success
174.203 test-42| Works: 1125
174.203 test-42| Fails: 1500
174.203 test-42| At:    1312
174.203 test-42| Message: mtu 1312 64
174.203 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53355
174.252 test-42| Got datagram of 1024 bytes.
174.252 test-42| Success
174.252 test-42| Works: 1312
174.252 test-42| Fails: 1500
174.252 test-42| At:    1406
174.252 test-42| Message: mtu 1406 64
174.253 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53356
174.301 test-42| Got datagram of 1024 bytes.
174.301 test-42| Success
174.301 test-42| Works: 1406
174.301 test-42| Fails: 1500
174.301 test-42| At:    1453
174.301 test-42| Message: mtu 1453 64
174.302 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53357
174.353 test-42| Got datagram of 1024 bytes.
174.354 test-42| Success
174.354 test-42| Works: 1453
174.354 test-42| Fails: 1500
174.354 test-42| At:    1476
174.354 test-42| Message: mtu 1476 64
174.354 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53358
174.408 test-42| Got datagram of 1024 bytes.
174.408 test-42| Success
174.408 test-42| Works: 1476
174.408 test-42| Fails: 1500
174.408 test-42| At:    1488
174.408 test-42| Message: mtu 1488 64
174.408 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53359
174.468 test-42| Got datagram of 31 bytes.
174.468 test-42| Responsive failure
174.468 test-42| Response is bad 1488 2001:470:0:90::2 1480
174.468 test-42| Works: 1476
174.468 test-42| Fails: 1488
174.468 test-42| At:    1482
174.468 test-42| Message: mtu 1482 64
174.469 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53360
174.528 test-42| Got datagram of 31 bytes.
174.528 test-42| Responsive failure
174.529 test-42| Response is bad 1482 2001:470:0:90::2 1480
174.529 test-42| Works: 1476
174.529 test-42| Fails: 1482
174.529 test-42| At:    1479
174.529 test-42| Message: mtu 1479 64
174.529 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53361
174.580 test-42| Got datagram of 1024 bytes.
174.580 test-42| Success
174.580 test-42| Works: 1479
174.580 test-42| Fails: 1482
174.580 test-42| At:    1480
174.580 test-42| Message: mtu 1480 64
174.580 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53362
174.628 test-42| Got datagram of 1024 bytes.
174.628 test-42| Success
174.628 test-42| Works: 1480
174.628 test-42| Fails: 1482
174.628 test-42| At:    1481
174.628 test-42| Message: mtu 1481 64
174.629 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53363
174.689 test-42| Got datagram of 31 bytes.
174.689 test-42| Responsive failure
174.689 test-42| Response is bad 1481 2001:470:0:90::2 1480
174.689 test-42| Final MTU is 1480

Wireshark Capture of the box running the client:

    230 174.728043  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      838    Source port: 53349  Destination port: eye2eye
    231 174.731075  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 TCP      86     sentinelsrm > 57537 [ACK] Seq=43 Ack=2 Win=5712 Len=0 TSval=87108372 TSecr=1298810232
    232 174.779641  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 ICMPv6   1294   Destination Unreachable (Port unreachable)
    233 174.780117  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      78     Source port: 53350  Destination port: eye2eye
    234 174.836455  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 ICMPv6   126    Destination Unreachable (Port unreachable)
    235 174.837125  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x451f2d01)
    236 174.837131  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    241 175.838153  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x22a0eb7a)
    242 175.838159  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    243 176.838659  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x39ec621f)
    244 176.838665  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    245 177.839024  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x24e7bdfe)
    246 177.839031  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    249 178.259659  2001:470:e07d:1:c62c:3ff:fe2c:78e0 2001:470:e07d:1::1    ICMPv6   149    Destination Unreachable (Port unreachable)
    253 178.261531  2001:470:e07d:1:c62c:3ff:fe2c:78e0 2001:470:e07d:1::1    ICMPv6   149    Destination Unreachable (Port unreachable)
    254 178.261558  2001:470:e07d:1:c62c:3ff:fe2c:78e0 2001:470:e07d:1::1    ICMPv6   149    Destination Unreachable (Port unreachable)
    255 178.261569  2001:470:e07d:1:c62c:3ff:fe2c:78e0 2001:470:e07d:1::1    ICMPv6   149    Destination Unreachable (Port unreachable)
    257 178.261591  2001:470:e07d:1:c62c:3ff:fe2c:78e0 2001:470:e07d:1::1    ICMPv6   149    Destination Unreachable (Port unreachable)
    258 178.840290  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x29ffd029)
    259 178.840296  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    260 179.841590  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x3d394354)
    261 179.841596  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    264 180.842345  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x77c2fbc1)
    265 180.842362  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    266 181.842854  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x2fa68044)
    267 181.842860  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    268 182.844119  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x3a598b70)
    269 182.844126  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    272 183.845585  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x61c80d48)
    273 183.845592  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    274 184.846852  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53352  Destination port: bcs-lmserver
    275 184.908422  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      93     Source port: bcs-lmserver  Destination port: 53352
    276 184.909101  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      72     Source port: 53353  Destination port: bcs-lmserver
    277 184.961213  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      764    Source port: bcs-lmserver  Destination port: 53353
    278 184.961756  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53354  Destination port: bcs-lmserver
    279 185.016423  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1139   Source port: bcs-lmserver  Destination port: 53354
    280 185.016967  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53355  Destination port: bcs-lmserver
    281 185.065661  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1326   Source port: bcs-lmserver  Destination port: 53355
    282 185.066323  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53356  Destination port: bcs-lmserver
    283 185.114773  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1420   Source port: bcs-lmserver  Destination port: 53356
    284 185.115388  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53357  Destination port: bcs-lmserver
    285 185.167136  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1467   Source port: bcs-lmserver  Destination port: 53357
    286 185.167749  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53358  Destination port: bcs-lmserver
    287 185.221257  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1490   Source port: bcs-lmserver  Destination port: 53358
    288 185.221967  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53359  Destination port: bcs-lmserver
    289 185.281867  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      93     Source port: bcs-lmserver  Destination port: 53359
    290 185.282656  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53360  Destination port: bcs-lmserver
    291 185.342016  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      93     Source port: bcs-lmserver  Destination port: 53360
    292 185.342637  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53361  Destination port: bcs-lmserver
    293 185.393482  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1493   Source port: bcs-lmserver  Destination port: 53361
    294 185.393965  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53362  Destination port: bcs-lmserver
    295 185.441881  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1494   Source port: bcs-lmserver  Destination port: 53362
    296 185.442379  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53363  Destination port: bcs-lmserver

[bicknell]

Now, with that data, I'm looking at what is going on.

First set of tests is sending from my client to the ICSI server UDP packets > 1500 bytes.

Code Select Expand


163.913 test-42| Testing the ability to send a large UDP packet (2000 bytes) over IPv6


It then tries to send from source ports 55348-55350, destination port 1948 (eye2eye in /etc/services):

Code Select Expand


163.913 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.913 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53348
163.914 test-42| Got exception java.net.SocketException: Message too long on UDP test
163.914 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.914 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53349
163.966 test-42| Got exception java.net.PortUnreachableException: ICMP Port Unreachable on UDP test
163.966 test-42| Can't send UDP fragments
163.966 test-42| Testing the ability to receive a large UDP packet (2000 bytes) over IPv6
163.966 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.966 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53350
164.023 test-42| Got exception java.net.PortUnreachableException: ICMP Port Unreachable on UDP test
164.023 test-42| Can't receive UDP fragments


"Got exception java.net.SocketException: Message too long on UDP test" makes me think there is a local OS/Java error/limit.  I also find it interesting the ICSI server is returning port unreachable on these test packets.

Let's look at the tcpdump for that traffic:

Code Select Expand


    226 174.727246  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1510   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x36cd5178)
    227 174.727256  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     622    IPv6 fragment (nxt=UDP (0x11) off=1448 id=0x36cd5178)
    228 174.727648  2001:470:e07d:1::1    2001:470:e07d:1:a833:b6aa:c711:ffa1 ICMPv6   1294   Packet Too Big
    229 174.728036  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x1bd690be)
    230 174.728043  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      838    Source port: 53349  Destination port: eye2eye
    232 174.779641  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 ICMPv6   1294   Destination Unreachable (Port unreachable)
    233 174.780117  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      78     Source port: 53350  Destination port: eye2eye
    234 174.836455  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 ICMPv6   126    Destination Unreachable (Port unreachable)
    235 174.837125  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x451f2d01)
    236 174.837131  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    241 175.838153  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x22a0eb7a)
    242 175.838159  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    243 176.838659  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x39ec621f)
    244 176.838665  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    245 177.839024  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x24e7bdfe)
    246 177.839031  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver


I can't find the source port 53348 packet, but it appears we pick up what might be three fragments from it.  We also see a packet too big message in the middle.  The port 53349-53351 packets all appear to go out with a packet plus a fragment, but it appears the ICSI server never receives them.  This looks like fragments are being filtered on the path from me to ICSI for some reason.  I'll note the fragments are 1294 bytes, I have no way to check the tunnel MTU in the Apple Time Capsule.  Could the time capsule be sending tunnel packets larger than HE will accept?

Next up, attempts to send from ports 53351 to the ICSI server.

Code Select Expand


164.023 test-42| Attempting to send a packet with
164.023 test-42| fragmentation of 2009 bytes
164.023 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
165.024 test-42| No data received.
165.024 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
166.025 test-42| No data received.
166.025 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
167.025 test-42| No data received.
167.025 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
168.026 test-42| No data received.
168.027 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
169.028 test-42| No data received.
169.028 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
170.028 test-42| No data received.
170.029 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
171.029 test-42| No data received.
171.029 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
172.030 test-42| No data received.
172.030 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
173.032 test-42| No data received.
173.032 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
174.033 test-42| No data received.
174.033 test-42| No reply back


And the packet capture:

Code Select Expand


    235 174.837125  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x451f2d01)
    236 174.837131  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    241 175.838153  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x22a0eb7a)
    242 175.838159  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    243 176.838659  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x39ec621f)
    244 176.838665  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    245 177.839024  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x24e7bdfe)
    246 177.839031  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    258 178.840290  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x29ffd029)
    259 178.840296  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    260 179.841590  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x3d394354)
    261 179.841596  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    264 180.842345  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x77c2fbc1)
    265 180.842362  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    266 181.842854  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x2fa68044)
    267 181.842860  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    268 182.844119  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x3a598b70)
    269 182.844126  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver
    272 183.845585  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      IPv6     1294   IPv6 fragment (nxt=UDP (0x11) off=0 id=0x61c80d48)
    273 183.845592  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      847    Source port: 53351  Destination port: bcs-lmserver


Now, this is more interesting to me.  The packets leave my server, but apparently never make it to the ICSI box.  It's possible just the fragments don't make it, which prevents reassembly, or it could be that none of these packets make it.   Same question as before, Could the time capsule be sending tunnel packets larger than HE will accept?

Now we get the testing from the ICSI server to my client:

Code Select Expand


174.033 test-42| Now looking for the receive MTU. Trying 1500 first
174.033 test-42| MSG: mtu 1500 64
174.033 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53352
174.095 test-42| Got datagram of 31 bytes.
174.095 test-42| Response is bad 1500 2001:470:0:90::2 1480
174.095 test-42| Path MTU is <1500B
174.095 test-42| Beginning binary search to find the path MTU
174.095 test-42| Works: 0
174.095 test-42| Fails: 1500
174.095 test-42| At:    750
174.095 test-42| Message: mtu 750 64
174.095 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53353
174.148 test-42| Got datagram of 702 bytes.
174.148 test-42| Success
174.148 test-42| Works: 750
174.148 test-42| Fails: 1500
174.148 test-42| At:    1125
174.148 test-42| Message: mtu 1125 64
174.148 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53354
174.203 test-42| Got datagram of 1024 bytes.
174.203 test-42| Success
174.203 test-42| Works: 1125
174.203 test-42| Fails: 1500
174.203 test-42| At:    1312
174.203 test-42| Message: mtu 1312 64
174.203 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53355
174.252 test-42| Got datagram of 1024 bytes.
174.252 test-42| Success
174.252 test-42| Works: 1312
174.252 test-42| Fails: 1500
174.252 test-42| At:    1406
174.252 test-42| Message: mtu 1406 64
174.253 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53356
174.301 test-42| Got datagram of 1024 bytes.
174.301 test-42| Success
174.301 test-42| Works: 1406
174.301 test-42| Fails: 1500
174.301 test-42| At:    1453
174.301 test-42| Message: mtu 1453 64
174.302 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53357
174.353 test-42| Got datagram of 1024 bytes.
174.354 test-42| Success
174.354 test-42| Works: 1453
174.354 test-42| Fails: 1500
174.354 test-42| At:    1476
174.354 test-42| Message: mtu 1476 64
174.354 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53358
174.408 test-42| Got datagram of 1024 bytes.
174.408 test-42| Success
174.408 test-42| Works: 1476
174.408 test-42| Fails: 1500
174.408 test-42| At:    1488
174.408 test-42| Message: mtu 1488 64
174.408 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53359
174.468 test-42| Got datagram of 31 bytes.
174.468 test-42| Responsive failure
174.468 test-42| Response is bad 1488 2001:470:0:90::2 1480
174.468 test-42| Works: 1476
174.468 test-42| Fails: 1488
174.468 test-42| At:    1482
174.468 test-42| Message: mtu 1482 64
174.469 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53360
174.528 test-42| Got datagram of 31 bytes.
174.528 test-42| Responsive failure
174.529 test-42| Response is bad 1482 2001:470:0:90::2 1480
174.529 test-42| Works: 1476
174.529 test-42| Fails: 1482
174.529 test-42| At:    1479
174.529 test-42| Message: mtu 1479 64
174.529 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53361
174.580 test-42| Got datagram of 1024 bytes.
174.580 test-42| Success
174.580 test-42| Works: 1479
174.580 test-42| Fails: 1482
174.580 test-42| At:    1480
174.580 test-42| Message: mtu 1480 64
174.580 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53362
174.628 test-42| Got datagram of 1024 bytes.
174.628 test-42| Success
174.628 test-42| Works: 1480
174.628 test-42| Fails: 1482
174.628 test-42| At:    1481
174.628 test-42| Message: mtu 1481 64
174.629 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53363
174.689 test-42| Got datagram of 31 bytes.
174.689 test-42| Responsive failure
174.689 test-42| Response is bad 1481 2001:470:0:90::2 1480
174.689 test-42| Final MTU is 1480


And the TCPDump

Code Select Expand


   274 184.846852  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53352  Destination port: bcs-lmserver
    275 184.908422  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      93     Source port: bcs-lmserver  Destination port: 53352
    276 184.909101  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      72     Source port: 53353  Destination port: bcs-lmserver
    277 184.961213  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      764    Source port: bcs-lmserver  Destination port: 53353
    278 184.961756  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53354  Destination port: bcs-lmserver
    279 185.016423  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1139   Source port: bcs-lmserver  Destination port: 53354
    280 185.016967  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53355  Destination port: bcs-lmserver
    281 185.065661  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1326   Source port: bcs-lmserver  Destination port: 53355
    282 185.066323  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53356  Destination port: bcs-lmserver
    283 185.114773  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1420   Source port: bcs-lmserver  Destination port: 53356
    284 185.115388  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53357  Destination port: bcs-lmserver
    285 185.167136  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1467   Source port: bcs-lmserver  Destination port: 53357
    286 185.167749  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53358  Destination port: bcs-lmserver
    287 185.221257  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1490   Source port: bcs-lmserver  Destination port: 53358
    288 185.221967  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53359  Destination port: bcs-lmserver
    289 185.281867  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      93     Source port: bcs-lmserver  Destination port: 53359
    290 185.282656  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53360  Destination port: bcs-lmserver
    291 185.342016  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      93     Source port: bcs-lmserver  Destination port: 53360
    292 185.342637  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53361  Destination port: bcs-lmserver
    293 185.393482  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1493   Source port: bcs-lmserver  Destination port: 53361
    294 185.393965  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53362  Destination port: bcs-lmserver
    295 185.441881  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      1494   Source port: bcs-lmserver  Destination port: 53362
    296 185.442379  2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93      UDP      73     Source port: 53363  Destination port: bcs-lmserver
    297 185.502727  2607:f740:b::f93      2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP      93     Source port: bcs-lmserver  Destination port: 53363


We see no fragments making it from ICSI to my system, illustrating the problem.

I tried a couple of other things.  I've been running all of these  tests on an OSX Lion client.  I tried disabling the firewall on the box to be sure it wasn't something with that, no change in the results.  I then decided to try a Windows 7 box on the same LAN to get a different version of Java and different OS quirks going.  It ran into issues with the windows firewall (generating a few pop ups), so I disabled the firewall and tried again.  Same result.  At least it's a consistent result across OSX and Windows.

One last test, turned off "Block incoming IPv6 connections" on my Time Capsule, basically disabling the firewall on that device.  This resulted in a slightly different message from the Netalyzr scan:

Your system can not send or receive fragmented traffic over IPv6.
The path between your network and our system supports an MTU of at least 1280 bytes. The path between our system and your network has an MTU of 1480 bytes. The bottleneck is at IP address 2001:470:0:90::2. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.

Bottom line?  It does appear that I cannot receive any IPv6 fragments over my tunnel/router, and that IPv6 fragmented I send out do not make it to their destination.

What I think would be an awesome next step is if someone from HE could run a tcpdump on the tunnel broker server for all packets to/from my IPv6 address range while I run the test, and then provide that to me.  I could then compare how things look on the other side of the tunnel, and probably have a better idea what's going on.  I now wonder if there are two different problems:

1) Mismatched tunnel MTU.  Neither the Apple device nor HE allow me to set it (or even view it!), so I think this is a strong possibility.

2) One or more devices in the middle is filtering IPv6 Fragments.

[kcochran]

We're 1480 on the tunnel interfaces.  I'll see if we can get an MTU option in the interface somewhere.  Options would likely be 1480, 1472, and 1280, unless anyone can think of any other useful common values.

[bicknell]

Did some more testing, this time with iPerf.

I can send 1432 byte UDP packets from a host on the Internet across my tunnel to my home box.  As soon as I go to a 1433 byte packet, I receive nothing at home.

The 1432 iperf from the server end looks like:

Code Select Expand


11:48:57.223794 IP6 (flowlabel 0xd1a2a, hlim 64, next-header UDP (17) payload length: 1440) ussenterprise.ufp.org.63767 > 2001:470:e07d:1:21d:7dff:fea3:66ae.8010: [udp sum ok] UDP, length 1432
11:48:57.234794 IP6 (flowlabel 0xd1a2a, hlim 64, next-header UDP (17) payload length: 1440) ussenterprise.ufp.org.63767 > 2001:470:e07d:1:21d:7dff:fea3:66ae.8010: [udp sum ok] UDP, length 1432
11:48:57.244795 IP6 (flowlabel 0xd1a2a, hlim 64, next-header UDP (17) payload length: 1440) ussenterprise.ufp.org.63767 > 2001:470:e07d:1:21d:7dff:fea3:66ae.8010: [udp sum ok] UDP, length 1432
11:48:57.255795 IP6 (flowlabel 0xd1a2a, hlim 64, next-header UDP (17) payload length: 1440) ussenterprise.ufp.org.63767 > 2001:470:e07d:1:21d:7dff:fea3:66ae.8010: [udp sum ok] UDP, length 1432


The 1433:

Code Select Expand



11:49:39.392331 IP6 (flowlabel 0xe004b, hlim 64, next-header Fragment (44) payload length: 1440) ussenterprise.ufp.org > 2001:470:e07d:1:21d:7dff:fea3:66ae: frag (0xc142ccd0:0|1432) 59968 > 8010: UDP, length 1433
11:49:39.392334 IP6 (flowlabel 0xe004b, hlim 64, next-header Fragment (44) payload length: 17) ussenterprise.ufp.org > 2001:470:e07d:1:21d:7dff:fea3:66ae: frag (0xc142ccd0:1432|9)
11:49:39.643317 IP6 (flowlabel 0xe004b, hlim 64, next-header Fragment (44) payload length: 1440) ussenterprise.ufp.org > 2001:470:e07d:1:21d:7dff:fea3:66ae: frag (0xc7f16651:0|1432) 59968 > 8010: UDP, length 1433
11:49:39.643320 IP6 (flowlabel 0xe004b, hlim 64, next-header Fragment (44) payload length: 17) ussenterprise.ufp.org > 2001:470:e07d:1:21d:7dff:fea3:66ae: frag (0xc7f16651:1432|9)
11:49:39.894314 IP6 (flowlabel 0xe004b, hlim 64, next-header Fragment (44) payload length: 1440) ussenterprise.ufp.org > 2001:470:e07d:1:21d:7dff:fea3:66ae: frag (0xf6422ca8:0|1432) 59968 > 8010: UDP, length 1433
11:49:39.894316 IP6 (flowlabel 0xe004b, hlim 64, next-header Fragment (44) payload length: 17) ussenterprise.ufp.org > 2001:470:e07d:1:21d:7dff:fea3:66ae: frag (0xf6422ca8:1432|9)
11:49:40.145311 IP6 (flowlabel 0xe004b, hlim 64, next-header Fragment (44) payload length: 1440) ussenterprise.ufp.org > 2001:470:e07d:1:21d:7dff:fea3:66ae: frag (0xee742ee6:0|1432) 59968 > 8010: UDP, length 1433
11:49:40.145314 IP6 (flowlabel 0xe004b, hlim 64, next-header Fragment (44) payload length: 17) ussenterprise.ufp.org > 2001:470:e07d:1:21d:7dff:fea3:66ae: frag (0xee742ee6:1432|9)


Neither fragment makes it to me.

Now, in the other direction, from home, across the tunnel, to a box on the Internet, the MTU is different.  1232 is the largest packet that makes it without fragmentation. (Seen from the Internet host end)

Code Select Expand


11:52:23.052771 IP6 (flowlabel 0x644bd, hlim 56, next-header UDP (17) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae.64362 > ussenterprise.ufp.org.8010: [udp sum ok] UDP, length 1232
11:52:23.060641 IP6 (flowlabel 0x644bd, hlim 56, next-header UDP (17) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae.64362 > ussenterprise.ufp.org.8010: [udp sum ok] UDP, length 1232
11:52:23.070635 IP6 (flowlabel 0x644bd, hlim 56, next-header UDP (17) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae.64362 > ussenterprise.ufp.org.8010: [udp sum ok] UDP, length 1232
11:52:23.082502 IP6 (flowlabel 0x644bd, hlim 56, next-header UDP (17) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae.64362 > ussenterprise.ufp.org.8010: [udp sum ok] UDP, length 1232
11:52:23.089872 IP6 (flowlabel 0x644bd, hlim 56, next-header UDP (17) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae.64362 > ussenterprise.ufp.org.8010: [udp sum ok] UDP, length 1232


Bumping up to 1233, we see the packets get fragmented and make it out:

Code Select Expand


11:52:52.782560 IP6 (flowlabel 0x0e4cc, hlim 56, next-header Fragment (44) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae > ussenterprise.ufp.org: frag (0xe48c9858:0|1232) 44552 > 8010: UDP, length 1233
11:52:52.786058 IP6 (flowlabel 0x0e4cc, hlim 56, next-header Fragment (44) payload length: 17) 2001:470:e07d:1:21d:7dff:fea3:66ae > ussenterprise.ufp.org: frag (0xe48c9858:1232|9)
11:52:52.792804 IP6 (flowlabel 0x0e4cc, hlim 56, next-header Fragment (44) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae > ussenterprise.ufp.org: frag (0xf29e788f:0|1232) 44552 > 8010: UDP, length 1233
11:52:52.797676 IP6 (flowlabel 0x0e4cc, hlim 56, next-header Fragment (44) payload length: 17) 2001:470:e07d:1:21d:7dff:fea3:66ae > ussenterprise.ufp.org: frag (0xf29e788f:1232|9)
11:52:52.807295 IP6 (flowlabel 0x0e4cc, hlim 56, next-header Fragment (44) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae > ussenterprise.ufp.org: frag (0xa93258d6:0|1232) 44552 > 8010: UDP, length 1233
11:52:52.807298 IP6 (flowlabel 0x0e4cc, hlim 56, next-header Fragment (44) payload length: 17) 2001:470:e07d:1:21d:7dff:fea3:66ae > ussenterprise.ufp.org: frag (0xa93258d6:1232|9)
11:52:52.812541 IP6 (flowlabel 0x0e4cc, hlim 56, next-header Fragment (44) payload length: 1240) 2001:470:e07d:1:21d:7dff:fea3:66ae > ussenterprise.ufp.org: frag (0xfd13fade:0|1232) 44552 > 8010: UDP, length 1233
11:52:52.815789 IP6 (flowlabel 0x0e4cc, hlim 56, next-header Fragment (44) payload length: 17) 2001:470:e07d:1:21d:7dff:fea3:66ae > ussenterprise.ufp.org: frag (0xfd13fade:1232|9)


From this, I reach the following conclusions:

• The Time Capsule has an outbound MTU of 1280 on the tunnel.  1232 bytes of data + 8 bytes UDP header, + 40 bytes IPv6 Header = 1280.
• The HE Tunnel has an outbound MTU of 1480 (1500 - 20 bytes 6in4 header).   1432 bytes of data + 8 bytes UDP header + 40 bytes IPv6 header = 1480.
• The path from home to the internet is fine.  I tried sending random packet sizes from 1200-4000 bytes, all made it, and all were properly fragmented based on the outbound MTU of 1280.
• Looking at the FreeBSD host cache on both the client and the server confirms these numbers, they show a 1280 byte and 1480 byte MTU respectively.
• The path from the Internet to me is broken.  Any packets larger than 1432 bytes (of data) result in zero packets reaching me.

Something in the path from the Internet to my home server is filtering all fragmented IPv6 packets.  Figuring it is very unlikely to be a router in the middle I'm left with the same two suspect locations.  Most likely the Apple Time Capsule is refusing to process all incoming fragmented packets.  Less likely but still possible is that HE on their tunnel broker servers has some configuration to drop all fragments (possibly in an attempt to protect the boxes from DDoS or similar) that is inadvertently killing all of my fragments as well.

HE Folks, now that I've narrowed it down and can reproduce with iperf, can we try you guys doing a tcpdump on the tunnel server to see if my fragmented packets are making it to the tunnel server, and/or being sent out down the tunnel?  If so, it's got to be the Time Capsule, if not, we'll have a different direction to go in.

[snarked]

How about 9k (9216 bytes) if HE has any paths into its tunnel servers which support jumbo frames?

Although my current path to HE does not (my colo provider is still at 1500), I do set my interfaces at 9216 and let PMTU discover whatever lower MTU is supported.  I've had no connectivity problems doing this.

[kasperd]

It does appear that I cannot receive any IPv6 fragments over my tunnel/router

While running some unrelated test I noticed that I was receiving fragmented IPv6 packets over the tunnel from HE. I was using the test on http://test-ipv6.com/, and it was sending packets that were too large to make it through the tunnel. PMTU discovery did work as intended, though the server behaved in a way that surprised me a bit. The TCP segment that had triggered the ICMP response was fragmented and I received a fragmented TCP packet through the tunnel. I would have expected TCP to split the data into smaller segments, but that was not what happened. Later TCP traffic did however use smaller segments.

So I know fragmented packets can make it through the tunnel.

Based on the results so far, the error message from netalyzer shows up with different software on the client host, the error shows up both when using the HE tunnel and when using 6to4. It seems the only thing that is still in common between all the cases where the error message is seen is netalyzer. And even the transcript from netalyzer suggests that the proper ICMPv6 packet to indicate that fragmentation is needed was sent and processed. Notice how the transcripts lists the MTU value from the ICMPv6 packet.

Though that MTU value shows up in the transcript, it does appear that it is completely ignored from that point forward. It does a binary search for the MTU and ignores what it was told by the routers.

Is there still any reason to think this is not a flaw in netalyzer?