Connection breaks down after ~2K data traffic

[Sixtus]

Hi all,

I have a strnage problem here:
One tunnel from my workstation (/64 net) routed thru an ipv6 router and one from my server (/48 net) to HE.
If I try to transfer a bigger file via scp or try to access a website running on my server, the traffic drops down to 0 after approx. 2K bytes data transfer.
Maybe I have a misconfiguration somewhere but I cannot find the point of failure.
Maybe someone tries to access http://commons.ipv6.tuxfutter.de/wiki/Main_Page. Here it loads and loads and loads....and some data is coming to my browser but it fails to load (bigger) pictures (like the logo of that wiki).

My configuration on my workstation:

Code Select Expand


eth0      Link encap:Ethernet  HWaddr 00:02:44:2b:5d:db
          inet addr:192.168.1.14  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: 2001:470:1f0b:cd0:202:44ff:fe2b:5ddb/64 Scope:Global
          inet6 addr: fe80::202:44ff:fe2b:5ddb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:90919 errors:0 dropped:0 overruns:0 frame:0
          TX packets:63558 errors:0 dropped:0 overruns:0 carrier:0
          collisions:663 txqueuelen:1000
          RX bytes:107822966 (102.8 MB)  TX bytes:6969994 (6.6 MB)
          Interrupt:20 Base address:0x3000

root@fafnir:~# ip -6 route show
2001:470:1f0b:cd0::/64 dev eth0  proto kernel  metric 256  expires 2592156sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev eth0  metric 256  expires 21323441sec mtu 1500 advmss 1440 hoplimit 4294967295
ff00::/8 dev eth0  metric 256  expires 21323441sec mtu 1500 advmss 1440 hoplimit 4294967295
default via fe80::250:fcff:fefa:624 dev eth0  proto kernel  metric 1024  expires 24sec mtu 1500 advmss 1440 hoplimit 64


Now my router (a Linux box):

Code Select Expand


hauke@athene:~$ ifconfig
eth1      Link encap:Ethernet  HWaddr 00:02:B3:97:D9:D6
          inet6 addr: fe80::202:b3ff:fe97:d9d6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:138447 errors:0 dropped:0 overruns:0 frame:0
          TX packets:113619 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:120981350 (115.3 MiB)  TX bytes:10384145 (9.9 MiB)

eth2      Link encap:Ethernet  HWaddr 00:50:FC:FA:06:24
          inet addr:192.168.1.13  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::250:fcff:fefa:624/64 Scope:Link
          inet6 addr: 2001:470:1f0b:cd0:2a0:c9ff:fef0:cbe/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:115657 errors:0 dropped:0 overruns:0 frame:0
          TX packets:142179 errors:0 dropped:0 overruns:0 carrier:0
          collisions:2543 txqueuelen:1000
          RX bytes:10484001 (9.9 MiB)  TX bytes:124987951 (119.1 MiB)
          Interrupt:11 Base address:0xf00

he-ipv6   Link encap:IPv6-in-IPv4
          inet6 addr: 2001:470:1f0a:cd0::2/64 Scope:Global
          inet6 addr: fe80::543d:62a8/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1472  Metric:1
          RX packets:1688 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1045 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1518449 (1.4 MiB)  TX bytes:165044 (161.1 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:4339 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4339 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1752295 (1.6 MiB)  TX bytes:1752295 (1.6 MiB)

ppp0      Link encap:Point-to-Point Protocol
          inet addr:84.61.98.168  P-t-P:84.61.96.1  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:138082 errors:0 dropped:0 overruns:0 frame:0
          TX packets:113228 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:117912082 (112.4 MiB)  TX bytes:7880869 (7.5 MiB)

hauke@athene:~$ ip -6 route show
2001:470:1f0a:cd0::/64 via :: dev he-ipv6  metric 256  expires 8529429sec mtu 1472 advmss 1412 hoplimit 4294967295
2001:470:1f0b:cd0::/64 dev eth2  metric 256  expires 8529285sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev eth1  metric 256  expires 8529284sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev eth2  metric 256  expires 8529284sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 via :: dev he-ipv6  metric 256  expires 8529429sec mtu 1472 advmss 1412 hoplimit 4294967295
ff00::/8 dev eth1  metric 256  expires 8529284sec mtu 1500 advmss 1440 hoplimit 4294967295
ff00::/8 dev eth2  metric 256  expires 8529284sec mtu 1500 advmss 1440 hoplimit 4294967295
ff00::/8 dev he-ipv6  metric 256  expires 8529429sec mtu 1472 advmss 1412 hoplimit 4294967295
default dev he-ipv6  metric 1024  expires 8529429sec mtu 1472 advmss 1412 hoplimit 4294967295


Now the configuration of the webserver:

Code Select Expand


eth0      Protokoll:Ethernet  Hardware Adresse 00:11:09:26:06:3D
          inet Adresse:217.172.178.228  Bcast:217.172.178.255  Maske:255.255.255.0
          inet6 Adresse: 2001:470:9b6c::1:1/48 Gültigkeitsbereich:Global
          inet6 Adresse: 2001:470:9b6c::11/48 Gültigkeitsbereich:Global
          inet6 Adresse: 2001:470:9b6c::1/48 Gültigkeitsbereich:Global
          inet6 Adresse: 2001:470:9b6c::1:3/48 Gültigkeitsbereich:Global
          inet6 Adresse: 2001:470:9b6c::2/48 Gültigkeitsbereich:Global
          inet6 Adresse: 2001:470:9b6c::1:2/48 Gültigkeitsbereich:Global
          inet6 Adresse: fe80::211:9ff:fe26:63d/64 Gültigkeitsbereich:Verbindung
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:103517201 errors:0 dropped:202279 overruns:0 frame:0
          TX packets:91491292 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:1000
          RX bytes:2085093570 (1.9 GiB)  TX bytes:1370013396 (1.2 GiB)
          Interrupt:11 Basisadresse:0xe500

he-ipv6   Protokoll:IPv6-nach-IPv4
          inet6 Adresse: fe80::d9ac:b2e4/128 Gültigkeitsbereich:Verbindung
          inet6 Adresse: 2001:470:1f0a:cd8::2/64 Gültigkeitsbereich:Global
          UP PUNKTZUPUNKT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:130142 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138819 errors:0 dropped:0 overruns:0 carrier:0
          Kollisionen:0 Sendewarteschlangenlänge:0
          RX bytes:97821857 (93.2 MiB)  TX bytes:59171503 (56.4 MiB)

[root@denver050:/etc/apache2]# ip -6 route show
default dev he-ipv6  metric 1024  mtu 1480 advmss 1420 hoplimit 4294967295



Any ideas?

Thanks and regards,
Sixtus

[jimb]

Sounds like the classic symptoms of "packet too big".  Remember that you're doing a 6in4 tunnel, so you're encapsulating your IPv6 in IPv4 on the way to HE, so the packets may get too big for one of the routers on the path.  It will be fine for smaller packets, such as small pings, ssh sessions, etc, but once TCP starts sending bigger packets, such as when it's transmitting picture files, bulk file transfers, etc, the packets may exceed the MTU of one of the router hops on the way back from HE to your 6in4 router, or in the IPv6 internet.  

Normally PMTUD (PMTUD) takes care of this problem, but it can be broken easily by one of the routers on the path, or by your firewall dropping certain types ICMP packets used in the process.

Lower the MTU of your 6in4 interface from 1480 to 1280.  This will cause the path MTU process to set PMTU on all your inside machines TCP/IPv6 stacks (your gateway will essentially tell your inside boxes to lower PMTU).  1280 should be a small enough number to get your tunnel traffic through to HE.  If not, you may need to adjust it down.

Normally this is an automatic process, but if one of the routers or your firewall is blocking ICMPv6 Packet too big (type 2, code 0) messages, then the PMTUD process will be broken.  You should probably also enable ICMPv4 Fragmentation needed packets (type 3, code 4) to pass to your gateway also, but I'm not sure if this even works for 6in4 (see below).

PMTUD is complicated in a tunneling situation because there are two levels at which it needs to work.  In the case of 6in4, the "first level" is the IPv4 6in4 tunnel traffic between your gateway and the peer gateway.  It could encounter MTU problems, and I'm honestly not sure if PMTUD works in this case.  If the 6in4 process/interface participates in PMTUD and adjusts down its MTU or route table MTUs then it would work.  Or if some process whereby the ICMPv4 Fragmentation needed message were passed along to, or acted on by IPv6 in some way, it would also work.  But AFAIK, this just has to be done "by hand" by using a low MTU on the 6in4 interface.

The "second level" is the IPv6 traffic itself, which could encounter MTU problems anywhere along the line, either while it's a passenger of IPv4 in the tunnel, or while on the IPv6 internet.  If it happens while in the "tunnel", again, I'm not sure how this is handled, since it is IPv4 traffic at this point.  But while in the IPv6 internet, it could also encounter MTU problems, in which case IPv6 PMTUD will take care of it.  In this case, PMTUD requires that ICMPv6 Packet Too Big messages (type 2, code 0) are received by the end nodes so that PMTU can be adjusted.  Note that this process can be broken if any router along the path drops the required ICMPv6 packets.

Anyway, long story short, change the MTU on your 6in4 interface to a lower #, 1280 seems to work for many people.  And also allow at least ICMPv6 PMTUD to work by allowing ICMPv6 Packet Too Big messages through end-to-end on your firewall.

[Sixtus]

Hi :)

Well, my answer took some days....

Anyway, I will try your hints.

Thanks and best,
Sixtus