Now, with that data, I'm looking at what is going on.
First set of tests is sending from my client to the ICSI server UDP packets > 1500 bytes.
163.913 test-42| Testing the ability to send a large UDP packet (2000 bytes) over IPv6
It then tries to send from source ports 55348-55350, destination port 1948 (eye2eye in /etc/services):
163.913 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.913 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53348
163.914 test-42| Got exception java.net.SocketException: Message too long on UDP test
163.914 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.914 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53349
163.966 test-42| Got exception java.net.PortUnreachableException: ICMP Port Unreachable on UDP test
163.966 test-42| Can't send UDP fragments
163.966 test-42| Testing the ability to receive a large UDP packet (2000 bytes) over IPv6
163.966 test-42| Sending UDP request to ipv6-node.u14369.n3.netalyzr.icsi.berkeley.edu on port 1948
163.966 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53350
164.023 test-42| Got exception java.net.PortUnreachableException: ICMP Port Unreachable on UDP test
164.023 test-42| Can't receive UDP fragments
"Got exception java.net.SocketException: Message too long on UDP test" makes me think there is a local OS/Java error/limit. I also find it interesting the ICSI server is returning port unreachable on these test packets.
Let's look at the tcpdump for that traffic:
226 174.727246 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1510 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x36cd5178)
227 174.727256 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 622 IPv6 fragment (nxt=UDP (0x11) off=1448 id=0x36cd5178)
228 174.727648 2001:470:e07d:1::1 2001:470:e07d:1:a833:b6aa:c711:ffa1 ICMPv6 1294 Packet Too Big
229 174.728036 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x1bd690be)
230 174.728043 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 838 Source port: 53349 Destination port: eye2eye
232 174.779641 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 ICMPv6 1294 Destination Unreachable (Port unreachable)
233 174.780117 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 78 Source port: 53350 Destination port: eye2eye
234 174.836455 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 ICMPv6 126 Destination Unreachable (Port unreachable)
235 174.837125 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x451f2d01)
236 174.837131 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
241 175.838153 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x22a0eb7a)
242 175.838159 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
243 176.838659 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x39ec621f)
244 176.838665 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
245 177.839024 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x24e7bdfe)
246 177.839031 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
I can't find the source port 53348 packet, but it appears we pick up what might be three fragments from it. We also see a packet too big message in the middle. The port 53349-53351 packets all appear to go out with a packet plus a fragment, but it appears the ICSI server never receives them. This looks like fragments are being filtered on the path from me to ICSI for some reason. I'll note the fragments are 1294 bytes, I have no way to check the tunnel MTU in the Apple Time Capsule.
Could the time capsule be sending tunnel packets larger than HE will accept?Next up, attempts to send from ports 53351 to the ICSI server.
164.023 test-42| Attempting to send a packet with
164.023 test-42| fragmentation of 2009 bytes
164.023 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
165.024 test-42| No data received.
165.024 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
166.025 test-42| No data received.
166.025 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
167.025 test-42| No data received.
167.025 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
168.026 test-42| No data received.
168.027 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
169.028 test-42| No data received.
169.028 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
170.028 test-42| No data received.
170.029 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
171.029 test-42| No data received.
171.029 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
172.030 test-42| No data received.
172.030 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
173.032 test-42| No data received.
173.032 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53351
174.033 test-42| No data received.
174.033 test-42| No reply back
And the packet capture:
235 174.837125 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x451f2d01)
236 174.837131 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
241 175.838153 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x22a0eb7a)
242 175.838159 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
243 176.838659 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x39ec621f)
244 176.838665 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
245 177.839024 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x24e7bdfe)
246 177.839031 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
258 178.840290 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x29ffd029)
259 178.840296 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
260 179.841590 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x3d394354)
261 179.841596 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
264 180.842345 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x77c2fbc1)
265 180.842362 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
266 181.842854 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x2fa68044)
267 181.842860 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
268 182.844119 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x3a598b70)
269 182.844126 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
272 183.845585 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 IPv6 1294 IPv6 fragment (nxt=UDP (0x11) off=0 id=0x61c80d48)
273 183.845592 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 847 Source port: 53351 Destination port: bcs-lmserver
Now, this is more interesting to me. The packets leave my server, but apparently never make it to the ICSI box. It's possible just the fragments don't make it, which prevents reassembly, or it could be that none of these packets make it. Same question as before,
Could the time capsule be sending tunnel packets larger than HE will accept?Now we get the testing from the ICSI server to my client:
174.033 test-42| Now looking for the receive MTU. Trying 1500 first
174.033 test-42| MSG: mtu 1500 64
174.033 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53352
174.095 test-42| Got datagram of 31 bytes.
174.095 test-42| Response is bad 1500 2001:470:0:90::2 1480
174.095 test-42| Path MTU is <1500B
174.095 test-42| Beginning binary search to find the path MTU
174.095 test-42| Works: 0
174.095 test-42| Fails: 1500
174.095 test-42| At: 750
174.095 test-42| Message: mtu 750 64
174.095 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53353
174.148 test-42| Got datagram of 702 bytes.
174.148 test-42| Success
174.148 test-42| Works: 750
174.148 test-42| Fails: 1500
174.148 test-42| At: 1125
174.148 test-42| Message: mtu 1125 64
174.148 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53354
174.203 test-42| Got datagram of 1024 bytes.
174.203 test-42| Success
174.203 test-42| Works: 1125
174.203 test-42| Fails: 1500
174.203 test-42| At: 1312
174.203 test-42| Message: mtu 1312 64
174.203 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53355
174.252 test-42| Got datagram of 1024 bytes.
174.252 test-42| Success
174.252 test-42| Works: 1312
174.252 test-42| Fails: 1500
174.252 test-42| At: 1406
174.252 test-42| Message: mtu 1406 64
174.253 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53356
174.301 test-42| Got datagram of 1024 bytes.
174.301 test-42| Success
174.301 test-42| Works: 1406
174.301 test-42| Fails: 1500
174.301 test-42| At: 1453
174.301 test-42| Message: mtu 1453 64
174.302 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53357
174.353 test-42| Got datagram of 1024 bytes.
174.354 test-42| Success
174.354 test-42| Works: 1453
174.354 test-42| Fails: 1500
174.354 test-42| At: 1476
174.354 test-42| Message: mtu 1476 64
174.354 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53358
174.408 test-42| Got datagram of 1024 bytes.
174.408 test-42| Success
174.408 test-42| Works: 1476
174.408 test-42| Fails: 1500
174.408 test-42| At: 1488
174.408 test-42| Message: mtu 1488 64
174.408 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53359
174.468 test-42| Got datagram of 31 bytes.
174.468 test-42| Responsive failure
174.468 test-42| Response is bad 1488 2001:470:0:90::2 1480
174.468 test-42| Works: 1476
174.468 test-42| Fails: 1488
174.468 test-42| At: 1482
174.468 test-42| Message: mtu 1482 64
174.469 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53360
174.528 test-42| Got datagram of 31 bytes.
174.528 test-42| Responsive failure
174.529 test-42| Response is bad 1482 2001:470:0:90::2 1480
174.529 test-42| Works: 1476
174.529 test-42| Fails: 1482
174.529 test-42| At: 1479
174.529 test-42| Message: mtu 1479 64
174.529 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53361
174.580 test-42| Got datagram of 1024 bytes.
174.580 test-42| Success
174.580 test-42| Works: 1479
174.580 test-42| Fails: 1482
174.580 test-42| At: 1480
174.580 test-42| Message: mtu 1480 64
174.580 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53362
174.628 test-42| Got datagram of 1024 bytes.
174.628 test-42| Success
174.628 test-42| Works: 1480
174.628 test-42| Fails: 1482
174.628 test-42| At: 1481
174.628 test-42| Message: mtu 1481 64
174.629 test-42| UDP socket at 2001:470:e07d:1:a833:b6aa:c711:ffa1%0:53363
174.689 test-42| Got datagram of 31 bytes.
174.689 test-42| Responsive failure
174.689 test-42| Response is bad 1481 2001:470:0:90::2 1480
174.689 test-42| Final MTU is 1480
And the TCPDump
274 184.846852 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53352 Destination port: bcs-lmserver
275 184.908422 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 93 Source port: bcs-lmserver Destination port: 53352
276 184.909101 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 72 Source port: 53353 Destination port: bcs-lmserver
277 184.961213 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 764 Source port: bcs-lmserver Destination port: 53353
278 184.961756 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53354 Destination port: bcs-lmserver
279 185.016423 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 1139 Source port: bcs-lmserver Destination port: 53354
280 185.016967 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53355 Destination port: bcs-lmserver
281 185.065661 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 1326 Source port: bcs-lmserver Destination port: 53355
282 185.066323 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53356 Destination port: bcs-lmserver
283 185.114773 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 1420 Source port: bcs-lmserver Destination port: 53356
284 185.115388 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53357 Destination port: bcs-lmserver
285 185.167136 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 1467 Source port: bcs-lmserver Destination port: 53357
286 185.167749 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53358 Destination port: bcs-lmserver
287 185.221257 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 1490 Source port: bcs-lmserver Destination port: 53358
288 185.221967 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53359 Destination port: bcs-lmserver
289 185.281867 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 93 Source port: bcs-lmserver Destination port: 53359
290 185.282656 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53360 Destination port: bcs-lmserver
291 185.342016 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 93 Source port: bcs-lmserver Destination port: 53360
292 185.342637 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53361 Destination port: bcs-lmserver
293 185.393482 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 1493 Source port: bcs-lmserver Destination port: 53361
294 185.393965 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53362 Destination port: bcs-lmserver
295 185.441881 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 1494 Source port: bcs-lmserver Destination port: 53362
296 185.442379 2001:470:e07d:1:a833:b6aa:c711:ffa1 2607:f740:b::f93 UDP 73 Source port: 53363 Destination port: bcs-lmserver
297 185.502727 2607:f740:b::f93 2001:470:e07d:1:a833:b6aa:c711:ffa1 UDP 93 Source port: bcs-lmserver Destination port: 53363
We see no fragments making it from ICSI to my system, illustrating the problem.
I tried a couple of other things. I've been running all of these tests on an OSX Lion client. I tried disabling the firewall on the box to be sure it wasn't something with that, no change in the results. I then decided to try a Windows 7 box on the same LAN to get a different version of Java and different OS quirks going. It ran into issues with the windows firewall (generating a few pop ups), so I disabled the firewall and tried again. Same result. At least it's a consistent result across OSX and Windows.
One last test, turned off "Block incoming IPv6 connections" on my Time Capsule, basically disabling the firewall on that device. This resulted in a slightly different message from the Netalyzr scan:
Your system can not send or receive fragmented traffic over IPv6.
The path between your network and our system supports an MTU of at least 1280 bytes. The path between our system and your network has an MTU of 1480 bytes. The bottleneck is at IP address 2001:470:0:90::2. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.
Bottom line? It does appear that I cannot receive any IPv6 fragments over my tunnel/router, and that IPv6 fragmented I send out do not make it to their destination.
What I think would be an awesome next step is if someone from HE could run a tcpdump on the tunnel broker server for all packets to/from my IPv6 address range while I run the test, and then provide that to me. I could then compare how things look on the other side of the tunnel, and probably have a better idea what's going on. I now wonder if there are two different problems:
1) Mismatched tunnel MTU. Neither the Apple device nor HE allow me to set it (or even view it!), so I think this is a strong possibility.
2) One or more devices in the middle is filtering IPv6 Fragments.