[SOLVED] IPv6 router not work on Ubuntu 12.04.2

[sirszabo]

Dear All,

I have a strange problem with my IPv6 configuration ...
In the last year I used the IPv6 tunneling on Debian 6.0 and everything worked fine. The IPv6 packets forwarded normally to the clients, etc.
About 2 weeks ago I changed the OS to Ubuntu 12.04.2. (unfortunately the Ubuntu development faster than the Debian  :-\ )
In the Ubuntu the routing not work anymore.

Here is my current configuration:
- OS: Ubuntu 12.04.2 (VMware virtual environment, server/router and the clients also)
- Kernel: 3.2.0-29-generic-pae
- VMware guest: 9.0.1-2

The Ubuntu router configuration:

/etc/network/interfaces

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
       pre-up modprobe ipv6
       endpoint 216.66.86.114
       local 79.xxx.xxx.xxx
       address 2001:470:xc:xxx::2
       netmask 64
       ttl 255
       up /sbin/ip -6 route add default dev he-ipv6
       up /sbin/ip -6 addr add 2001:470:xd:xxx::1/64 dev eth0 -> gateway IP
       up /sbin/ip -6 addr add 2001:470:xd:xxx::5/64 dev eth0 -> static IPv6 address for the server/router machine
       pre-down /sbin/ip -6 addr del 2001:470:xd:xxx::1/64 dev eth0
       pre-down /sbin/ip -6 addr del 2001:470:xd:xxx::5/64 dev eth0
       pre-down /sbin/ip -6 route del default dev he-ipv6

sysctl -p
net.ipv4.ip_forward = 1 (this is enabled for the OpenVPN)
net.ipv6.conf.all.forwarding = 1 (this is enabled for IPv6)

The default ufw firewall has been removed from the system and the iptables flushed, so no firewall.

Now from the IPv6 server/router work everything ... ping, ssh, web, mail etc.

wget test from router:

wget -O/dev/null http://ipv6.google.com/
--2013-03-21 16:52:05--  http://ipv6.google.com/
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:4016:801::1014
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:4016:801::1014|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/dev/null'

Ping test from router:

ping6 -c 4 ipv6.google.com
PING ipv6.google.com(muc03s02-in-x10.1e100.net) 56 data bytes
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=1 ttl=55 time=61.8 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=2 ttl=55 time=60.9 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=3 ttl=55 time=61.8 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=4 ttl=55 time=62.2 ms

--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3098ms
rtt min/avg/max/mdev = 60.980/61.730/62.246/0.462 ms

So, this is okay!

A Ubuntu client configuration:

/etc/network/interfaces

iface eth0 inet6 static
       pre-up modprobe ipv6
       address 2001:470:xd:xxx::4
       netmask 64
       gateway 2001:470:xd:xxx::1

wget test from client:

wget -O/dev/null http://ipv6.google.com/
--2013-03-21 16:51:58--  http://ipv6.google.com/
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:4016:801::1014
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:4016:801::1014|:80... failed: Connection timed out.
Retrying.

Ping test from a client:

ping6 -c 4 ipv6.google.com
PING ipv6.google.com(muc03s02-in-x10.1e100.net) 56 data bytes
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=1 ttl=54 time=74.6 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=2 ttl=54 time=74.4 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=3 ttl=54 time=74.7 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=4 ttl=54 time=74.3 ms

--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 74.395/74.596/74.797/0.159 ms

Just the ping are okay in the clients. Another communication are not possible. The strange, that this normally worked on the Debian 6 ...
Could you please help for me how I can restore the IPv6 communication in the Ubuntu environments?

Many thanks!

[cholzhauer]

That really sounds like a firewall issue to me.  You're not using your routed /48 are you?

[sirszabo]

Hello cholzhauer,

Thanks for your feedback!
I am same think, that this is fw issue or kernel (maybe more sysctl parameter necessary). Therefore I flushed the iptables and ip6tables also. Now I just enabled the ssh and ddos-attack rules in the IPv4.

IPv4 iptables:

iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ssh-ddos  tcp  --  anywhere             anywhere             multiport dports ssh
ssh  tcp  --  anywhere             anywhere             multiport dports ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain ssh (1 references)
target     prot opt source               destination
DROP       all  --  spiceafrica.com      anywhere
RETURN     all  --  anywhere             anywhere

Chain ssh-ddos (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

IPv6 iptables:

ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I just use the /64 routed IPv6 subnet, the /48 right now not in use. Do you have any idea, what can I do or what is need to check (maybe double)?
I already search this problem in google and of course write a mail to the HE support team. The support team can't help, because the router are okay and the communication working with the HE endpoint.

Thanks for your feedback!

[cholzhauer]

What if you just disable IPTables and try it then?

[kasperd]

I'd like to give some advice on how to get to the bottom of this issue, but there is just not enough information in this thread for me to figure out anything. For a starter you should tell us the IPv6 address of one of the unreachable hosts, such that we can try to do a traceroute towards it. Additionally, you should post output from a traceroute from one of those hosts towards any IPv6 address outside of the HE network.

Finally, the iptables rules you listed are not in a very useful format. The format you are using does not provide all the relevant details about the rules. The output from iptables-save and ip6tables-save is better as it does not leave out relevant details.

[sirszabo]

Thanks for your feedback!

Here is my complete details: (I modified some thinks since my last post)

HE configuration:

Code Select Expand


Server IPv4 Address:216.66.86.114
Server IPv6 Address:2001:470:xc:xxx::1/64
Client IPv4 Address:79.120.xxx.xxx
Client IPv6 Address:2001:470:xc:xxx::2/64
Routed /64:2001:470:xd:xxx::/64
Routed /48:2001:470:xxxx::/48


Router:

IP configuration on eth0

Code Select Expand



eth0    Link encap:Ethernet  HWaddr 00:50:56:00:01:00
         inet addr:79.120.xxx.xxx  Bcast:79.120.xxx.xxx  Mask:255.255.255.0
         inet6 addr: fe80::250:56ff:fe00:100/64 Scope:Link
         inet6 addr: 2001:470:xd:xxx::5/64 Scope:Global
         inet6 addr: 2001:470:xd:xxx::1/64 Scope:Global
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:94541 errors:0 dropped:58 overruns:0 frame:0
         TX packets:31173 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:29652862 (29.6 MB)  TX bytes:5728363 (5.7 MB)
         Interrupt:19 Base address:0x14a4



/etc/network/interfaces

Code Select Expand



iface eth0 inet6 static
       up /sbin/ifconfig eth0 inet6 add 2001:470:xd:xxx::5/64
       address 2001:470:xd:xxx::1
       netmask 64
       dns-nameservers 2001:470:xd:xxx::5 2001:470:xd:xxx::6

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
       endpoint 216.66.86.114
       local 79.120.xxx.xxx
       address 2001:470:xc:xxx::2
       netmask 64
       ttl 255
       gateway 2001:470:xc:xxx::1
       up ip -6 route add ::/0 dev he-ipv6



sysctl parameters:

Code Select Expand



net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 1



iptables output:

Code Select Expand



# Generated by iptables-save v1.4.12 on Sun Mar 24 17:26:47 2013
*filter
:INPUT ACCEPT [43574:25075608]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [26600:4794387]
:ssh - [0:0]
:ssh-ddos - [0:0]
-A INPUT -p tcp -m multiport --dports 22 -j ssh-ddos
-A INPUT -p tcp -m multiport --dports 22 -j ssh
-A ssh -j RETURN
-A ssh-ddos -j RETURN
COMMIT
# Completed on Sun Mar 24 17:26:47 2013



ip6tables output:

Code Select Expand



# Generated by ip6tables-save v1.4.12 on Sun Mar 24 17:28:00 2013
*filter
:INPUT ACCEPT [14805:19590286]
:FORWARD ACCEPT [723:81226]
:OUTPUT ACCEPT [5269:484438]
COMMIT
# Completed on Sun Mar 24 17:28:00 2013



ping test from the router:

Code Select Expand



ping6 -c 4 ipv6.google.com
PING ipv6.google.com(muc03s01-in-x14.1e100.net) 56 data bytes
64 bytes from muc03s01-in-x14.1e100.net: icmp_seq=1 ttl=55 time=61.6 ms
64 bytes from muc03s01-in-x14.1e100.net: icmp_seq=2 ttl=55 time=61.4 ms
64 bytes from muc03s01-in-x14.1e100.net: icmp_seq=3 ttl=55 time=62.0 ms
64 bytes from muc03s01-in-x14.1e100.net: icmp_seq=4 ttl=55 time=61.5 ms

--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3007ms
rtt min/avg/max/mdev = 61.472/61.667/62.014/0.367 ms



traceroute from the router:

Code Select Expand



traceroute6 ipv6.google.com
traceroute to ipv6.l.google.com (2a00:1450:4016:800::1014) from 2001:470:xc:xxx::2, 30 hops max, 16 byte packets
1  xxxxxxxxx.tunnel.tserv26.ber1.ipv6.he.net (2001:470:xc:xxx::1)  42.284 ms  43.324 ms  42.098 ms
2  gige-g2-20.core1.ber1.he.net (2001:470:0:220::1)  41.174 ms  40.775 ms  40.677 ms
3  2001:7f8:19:1::3b41:1 (2001:7f8:19:1::3b41:1)  66.83 ms  67.463 ms  67.647 ms
4  2001:4860::1:0:60d (2001:4860::1:0:60d)  64.387 ms  59.54 ms  59.337 ms
5  2001:4860::8:0:3098 (2001:4860::8:0:3098)  59.162 ms  58.728 ms  59.396 ms
6  2001:4860::1:0:336c (2001:4860::1:0:336c)  62.234 ms  62.359 ms  62.598 ms
7  2001:4860:0:1::535 (2001:4860:0:1::535)  62.727 ms  63.296 ms  62.968 ms
8  2a00:1450:8000:1e::8 (2a00:1450:8000:1e::8)  71.093 ms  68.595 ms  66.701 ms



wget test from the router:

Code Select Expand



wget -O/dev/null http://ipv6.google.com/
--2013-03-24 17:30:40--  http://ipv6.google.com/
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:4016:800::1014
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:4016:800::1014|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/dev/null'

   [ <=>                                                                                                                                                        ] 10,800      --.-K/s   in 0s

2013-03-24 17:30:40 (81.3 MB/s) - `/dev/null' saved [10800]



IPv6 routing from the router:

Code Select Expand



ip -6 route show
2001:470:xc:xxx::1 dev he-ipv6  metric 1024
2001:470:xc:xxx::/64 via :: dev he-ipv6  proto kernel  metric 256
2001:470:xd:xxx::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev eth1  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
fe80::/64 dev tap0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev eth2  proto kernel  metric 256
fe80::/64 via :: dev he-ipv6  proto kernel  metric 256
default via 2001:470:xc:xxx::1 dev he-ipv6  metric 1024
default dev he-ipv6  metric 1024



Client configuration:

IP configuration on eth0

Code Select Expand



eth0    Link encap:Ethernet  HWaddr 00:50:56:00:02:00
         inet addr:195.184.xxx.xxx  Bcast:195.184.xxx.xxx  Mask:255.255.255.0
         inet6 addr: fe80::250:56ff:fe00:200/64 Scope:Link
         inet6 addr: 2001:470:xd:xxx::6/64 Scope:Global
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:76170 errors:0 dropped:8 overruns:0 frame:0
         TX packets:12191 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:44070379 (44.0 MB)  TX bytes:2137837 (2.1 MB)
         Interrupt:19 Base address:0x14a4



/etc/network/interfaces

Code Select Expand



iface eth0 inet6 static
       address 2001:470:xd:xxx::6
       netmask 64
       gateway 2001:470:xd:xxx::1
       dns-nameservers 2001:470:xd:xxx::5 2001:470:xd:xxx::6



no sysctl parameters.

iptables output

Code Select Expand



# Generated by iptables-save v1.4.12 on Sun Mar 24 17:37:53 2013
*filter
:INPUT ACCEPT [33599:39575179]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [7271:1209951]
:ssh - [0:0]
:ssh-ddos - [0:0]
-A INPUT -p tcp -m multiport --dports 22 -j ssh-ddos
-A INPUT -p tcp -m multiport --dports 22 -j ssh
-A ssh -s 63.141.228.50/32 -j DROP
-A ssh -j RETURN
-A ssh-ddos -j RETURN
COMMIT
# Completed on Sun Mar 24 17:37:53 2013



ip6tables output:

Code Select Expand



# Generated by ip6tables-save v1.4.12 on Sun Mar 24 17:39:18 2013
*filter
:INPUT ACCEPT [1142:128841]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2207:193460]
COMMIT
# Completed on Sun Mar 24 17:39:18 2013



ping test from the client:

Code Select Expand



ping6 -c 4 ipv6.google.com
PING ipv6.google.com(muc03s01-in-x12.1e100.net) 56 data bytes
64 bytes from muc03s01-in-x12.1e100.net: icmp_seq=1 ttl=54 time=63.3 ms
64 bytes from muc03s01-in-x12.1e100.net: icmp_seq=2 ttl=54 time=63.6 ms
64 bytes from muc03s01-in-x12.1e100.net: icmp_seq=3 ttl=54 time=63.0 ms
64 bytes from muc03s01-in-x12.1e100.net: icmp_seq=4 ttl=54 time=62.5 ms

--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3008ms
rtt min/avg/max/mdev = 62.552/63.142/63.602/0.390 ms



traceroute from the client:

Code Select Expand



traceroute6 ipv6.google.com
traceroute to ipv6.l.google.com (2a00:1450:4016:800::1012) from 2001:470:xd:xxx::6, 30 hops max, 16 byte packets
1  xxxx.xxxxxxx.hu (2001:470:xd:xxx::5)  0.56 ms  0.277 ms  0.171 ms
2  xxxx.tunnel.tserv26.ber1.ipv6.he.net (2001:470:xc:xxx::1)  43.068 ms  42.747 ms  42.109 ms
3  gige-g2-20.core1.ber1.he.net (2001:470:0:220::1)  41.343 ms  40.818 ms  41.344 ms
4  2001:7f8:19:1::3b41:1 (2001:7f8:19:1::3b41:1)  65.077 ms  61.184 ms  59.244 ms
5  2001:4860::1:0:60d (2001:4860::1:0:60d)  59.202 ms  58.63 ms  60.596 ms
6  2001:4860::8:0:3098 (2001:4860::8:0:3098)  59.333 ms  60.176 ms  59.824 ms
7  2001:4860::1:0:336c (2001:4860::1:0:336c)  65.093 ms  62.863 ms  62.805 ms
8  2001:4860:0:1::535 (2001:4860:0:1::535)  62.896 ms  64.405 ms  62.897 ms
9  2a00:1450:8000:1e::1 (2a00:1450:8000:1e::1)  68.002 ms  64.734 ms  64.679 ms



wget test from the client:

Code Select Expand



wget -O/dev/null http://ipv6.google.com/
--2013-03-24 17:44:26--  http://ipv6.google.com/
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:4016:800::1012
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:4016:800::1012|:80... failed: Connection timed out.
Retrying.



IPv6 routing from the client:

Code Select Expand



ip -6 route show
2001:470:xd:xxx::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev eth1  proto kernel  metric 256
fe80::/64 dev eth2  proto kernel  metric 256
default via 2001:470:xd:xxx::1 dev eth0  metric 1024



The router and client IPv4 address are different, but all on the same (physical) router. As I see the router not forward any packets to the clients. I already tested the traceroute via http://www.subnetonline.com/pages/ipv6-network-tools/online-ipv6-traceroute.php. BOTH IPv6 was okay.

- The IPv4 and IPv6 addresses has been removed after the problem solved!! -

[kasperd]

I couldn't spot any mistakes in your configuration. Running the tunnel and the IPv6 LAN on the same interface stroke me as unusual. But I believe that is correct in your setup. Besides, if that was the source of the problem, the symptoms would look different.

I tried a few variations of traceroute commands towards your addresses, which mostly confirmed what you already observed. But then I got a clue from using traceroute6 with TCP packets.

A telnet command from my computer to 2001:470:6d:318::6 port 80 times out. But a traceroute6 commands with TCP packets send to the exact same port showed success.

Why would two tools testing the exact same thing disagree like that?

I fired up tcpdump and I could see my telnet command causing my kernel to send TCP SYN packets, which 2001:470:6d:318::6 replied to with TCP RST packets. The kernel on my computer just ignored the TCP RST packets and kept retransmitting the SYN packet.

Why would the kernel ignore a TCP RST packet? The first thing that came to my mind was possibly an invalid TCP checksum. So I fired up wireshark to look more closely on the TCP RST packets from your host. Lo and behold, the TCP RST packets did indeed have invalid checksums. I saw checksum B1DD on a couple of TCP RST packets, that should have had checksums 0300 and 64A0.

If the packets had correct checksums when they were sent and got corrupted in transit, it would be unlikely for them to show up at my end with identical checksums. Which leads me to believe they had invalid checksum when they left 2001:470:6d:318::6 in the first place.

So first thing you need to check is if 2001:470:6d:318::6 is running a buggy kernel with a faulty TCPv6 stack. Can you open TCP connections between 2001:470:6d:318::6 and 2001:470:6d:318::1? Have you tried running ssh over IPv6 between them? And are those two hosts running the same kernel version?

If the faulty TCP checksum is not caused by the kernel version, it might have to do with TCP checksum offloading. Some network interfaces can do checksumming in hardware, so the TCP stack just hands the packet to the network interface with an invalid checksum and tells the network interface to do the calculation.

As I understand it, in your case the network interface causing your trouble is a virtual network interface. Could it be that it is emulating a particular piece of hardware with support for TCP checksum offloading on both IPv4 and IPv6, but the emulation of said hardware is incomplete and supports only TCP checksum offloading on IPv4? If that's the case, it means the drive is strictly speaking not compatible with the emulation.

I am just guessing at possible reasons, but if it turns out to really be the case that the problem is caused by kernel using TCP checksum offloading on an emulated network interface, which does not support it, there are three ways it could be solved.

• Upgrade driver to a later version, which may know about this subtle difference between the real network interface and the emulated one.
• Upgrade VMWare to a later version with more complete hardware emulation.
• Just turn off the usage of TCP checksum offloading to let the kernel do it before handing the packet to the network interface.

[kasperd]

Looks like you got it working. What did you need to change to get correct checksums on the TCP packets?

[sirszabo]

Dear kasperd,

Many-many thanks for your help and hints! The problem has been solved as you wrote me the 3rd option. I switched off the ChecksumOffload in the vmware servers. (than reboot the all virtual machines) Everything working normally. Packets are okay and my VPN clients same receive the IPV6 address. So, everything restored back to the normal state.
I am very thankful for your analysis/help and of course your time!!!!

Have a nice day, and many thanks again!!!