Dear All,
I have a strange problem with my IPv6 configuration ...
In the last year I used the IPv6 tunneling on Debian 6.0 and everything worked fine. The IPv6 packets forwarded normally to the clients, etc.
About 2 weeks ago I changed the OS to Ubuntu 12.04.2. (unfortunately the Ubuntu development faster than the Debian :-\ )
In the Ubuntu the routing not work anymore.
Here is my current configuration:
- OS: Ubuntu 12.04.2 (VMware virtual environment, server/router and the clients also)
- Kernel: 3.2.0-29-generic-pae
- VMware guest: 9.0.1-2
The Ubuntu router configuration:
/etc/network/interfaces
auto he-ipv6
iface he-ipv6 inet6 v4tunnel
pre-up modprobe ipv6
endpoint 216.66.86.114
local 79.xxx.xxx.xxx
address 2001:470:xc:xxx::2
netmask 64
ttl 255
up /sbin/ip -6 route add default dev he-ipv6
up /sbin/ip -6 addr add 2001:470:xd:xxx::1/64 dev eth0 -> gateway IP
up /sbin/ip -6 addr add 2001:470:xd:xxx::5/64 dev eth0 -> static IPv6 address for the server/router machine
pre-down /sbin/ip -6 addr del 2001:470:xd:xxx::1/64 dev eth0
pre-down /sbin/ip -6 addr del 2001:470:xd:xxx::5/64 dev eth0
pre-down /sbin/ip -6 route del default dev he-ipv6
sysctl -p
net.ipv4.ip_forward = 1 (this is enabled for the OpenVPN)
net.ipv6.conf.all.forwarding = 1 (this is enabled for IPv6)
The default ufw firewall has been removed from the system and the iptables flushed, so no firewall.
Now from the IPv6 server/router work everything ... ping, ssh, web, mail etc.
wget test from router:
wget -O/dev/null http://ipv6.google.com/
--2013-03-21 16:52:05-- http://ipv6.google.com/
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:4016:801::1014
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:4016:801::1014|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/dev/null'
Ping test from router:
ping6 -c 4 ipv6.google.com
PING ipv6.google.com(muc03s02-in-x10.1e100.net) 56 data bytes
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=1 ttl=55 time=61.8 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=2 ttl=55 time=60.9 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=3 ttl=55 time=61.8 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=4 ttl=55 time=62.2 ms
--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3098ms
rtt min/avg/max/mdev = 60.980/61.730/62.246/0.462 ms
So, this is okay!
A Ubuntu client configuration:
/etc/network/interfaces
iface eth0 inet6 static
pre-up modprobe ipv6
address 2001:470:xd:xxx::4
netmask 64
gateway 2001:470:xd:xxx::1
wget test from client:
wget -O/dev/null http://ipv6.google.com/
--2013-03-21 16:51:58-- http://ipv6.google.com/
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:4016:801::1014
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:4016:801::1014|:80... failed: Connection timed out.
Retrying.
Ping test from a client:
ping6 -c 4 ipv6.google.com
PING ipv6.google.com(muc03s02-in-x10.1e100.net) 56 data bytes
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=1 ttl=54 time=74.6 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=2 ttl=54 time=74.4 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=3 ttl=54 time=74.7 ms
64 bytes from muc03s02-in-x10.1e100.net: icmp_seq=4 ttl=54 time=74.3 ms
--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 74.395/74.596/74.797/0.159 ms
Just the ping are okay in the clients. Another communication are not possible. The strange, that this normally worked on the Debian 6 ...
Could you please help for me how I can restore the IPv6 communication in the Ubuntu environments?
Many thanks!
That really sounds like a firewall issue to me. You're not using your routed /48 are you?
Hello cholzhauer,
Thanks for your feedback!
I am same think, that this is fw issue or kernel (maybe more sysctl parameter necessary). Therefore I flushed the iptables and ip6tables also. Now I just enabled the ssh and ddos-attack rules in the IPv4.
IPv4 iptables:
iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ssh-ddos tcp -- anywhere anywhere multiport dports ssh
ssh tcp -- anywhere anywhere multiport dports ssh
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain ssh (1 references)
target prot opt source destination
DROP all -- spiceafrica.com anywhere
RETURN all -- anywhere anywhere
Chain ssh-ddos (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
IPv6 iptables:
ip6tables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
I just use the /64 routed IPv6 subnet, the /48 right now not in use. Do you have any idea, what can I do or what is need to check (maybe double)?
I already search this problem in google and of course write a mail to the HE support team. The support team can't help, because the router are okay and the communication working with the HE endpoint.
Thanks for your feedback!
What if you just disable IPTables and try it then?
I'd like to give some advice on how to get to the bottom of this issue, but there is just not enough information in this thread for me to figure out anything. For a starter you should tell us the IPv6 address of one of the unreachable hosts, such that we can try to do a traceroute towards it. Additionally, you should post output from a traceroute from one of those hosts towards any IPv6 address outside of the HE network.
Finally, the iptables rules you listed are not in a very useful format. The format you are using does not provide all the relevant details about the rules. The output from iptables-save and ip6tables-save is better as it does not leave out relevant details.
Thanks for your feedback!
Here is my complete details: (I modified some thinks since my last post)
HE configuration:
Server IPv4 Address:216.66.86.114
Server IPv6 Address:2001:470:xc:xxx::1/64
Client IPv4 Address:79.120.xxx.xxx
Client IPv6 Address:2001:470:xc:xxx::2/64
Routed /64:2001:470:xd:xxx::/64
Routed /48:2001:470:xxxx::/48
Router:
IP configuration on eth0
eth0 Link encap:Ethernet HWaddr 00:50:56:00:01:00
inet addr:79.120.xxx.xxx Bcast:79.120.xxx.xxx Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe00:100/64 Scope:Link
inet6 addr: 2001:470:xd:xxx::5/64 Scope:Global
inet6 addr: 2001:470:xd:xxx::1/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:94541 errors:0 dropped:58 overruns:0 frame:0
TX packets:31173 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:29652862 (29.6 MB) TX bytes:5728363 (5.7 MB)
Interrupt:19 Base address:0x14a4
/etc/network/interfaces
iface eth0 inet6 static
up /sbin/ifconfig eth0 inet6 add 2001:470:xd:xxx::5/64
address 2001:470:xd:xxx::1
netmask 64
dns-nameservers 2001:470:xd:xxx::5 2001:470:xd:xxx::6
auto he-ipv6
iface he-ipv6 inet6 v4tunnel
endpoint 216.66.86.114
local 79.120.xxx.xxx
address 2001:470:xc:xxx::2
netmask 64
ttl 255
gateway 2001:470:xc:xxx::1
up ip -6 route add ::/0 dev he-ipv6
sysctl parameters:
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 1
iptables output:
# Generated by iptables-save v1.4.12 on Sun Mar 24 17:26:47 2013
*filter
:INPUT ACCEPT [43574:25075608]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [26600:4794387]
:ssh - [0:0]
:ssh-ddos - [0:0]
-A INPUT -p tcp -m multiport --dports 22 -j ssh-ddos
-A INPUT -p tcp -m multiport --dports 22 -j ssh
-A ssh -j RETURN
-A ssh-ddos -j RETURN
COMMIT
# Completed on Sun Mar 24 17:26:47 2013
ip6tables output:
# Generated by ip6tables-save v1.4.12 on Sun Mar 24 17:28:00 2013
*filter
:INPUT ACCEPT [14805:19590286]
:FORWARD ACCEPT [723:81226]
:OUTPUT ACCEPT [5269:484438]
COMMIT
# Completed on Sun Mar 24 17:28:00 2013
ping test from the router:
ping6 -c 4 ipv6.google.com
PING ipv6.google.com(muc03s01-in-x14.1e100.net) 56 data bytes
64 bytes from muc03s01-in-x14.1e100.net: icmp_seq=1 ttl=55 time=61.6 ms
64 bytes from muc03s01-in-x14.1e100.net: icmp_seq=2 ttl=55 time=61.4 ms
64 bytes from muc03s01-in-x14.1e100.net: icmp_seq=3 ttl=55 time=62.0 ms
64 bytes from muc03s01-in-x14.1e100.net: icmp_seq=4 ttl=55 time=61.5 ms
--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3007ms
rtt min/avg/max/mdev = 61.472/61.667/62.014/0.367 ms
traceroute from the router:
traceroute6 ipv6.google.com
traceroute to ipv6.l.google.com (2a00:1450:4016:800::1014) from 2001:470:xc:xxx::2, 30 hops max, 16 byte packets
1 xxxxxxxxx.tunnel.tserv26.ber1.ipv6.he.net (2001:470:xc:xxx::1) 42.284 ms 43.324 ms 42.098 ms
2 gige-g2-20.core1.ber1.he.net (2001:470:0:220::1) 41.174 ms 40.775 ms 40.677 ms
3 2001:7f8:19:1::3b41:1 (2001:7f8:19:1::3b41:1) 66.83 ms 67.463 ms 67.647 ms
4 2001:4860::1:0:60d (2001:4860::1:0:60d) 64.387 ms 59.54 ms 59.337 ms
5 2001:4860::8:0:3098 (2001:4860::8:0:3098) 59.162 ms 58.728 ms 59.396 ms
6 2001:4860::1:0:336c (2001:4860::1:0:336c) 62.234 ms 62.359 ms 62.598 ms
7 2001:4860:0:1::535 (2001:4860:0:1::535) 62.727 ms 63.296 ms 62.968 ms
8 2a00:1450:8000:1e::8 (2a00:1450:8000:1e::8) 71.093 ms 68.595 ms 66.701 ms
wget test from the router:
wget -O/dev/null http://ipv6.google.com/
--2013-03-24 17:30:40-- http://ipv6.google.com/
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:4016:800::1014
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:4016:800::1014|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/dev/null'
[ <=> ] 10,800 --.-K/s in 0s
2013-03-24 17:30:40 (81.3 MB/s) - `/dev/null' saved [10800]
IPv6 routing from the router:
ip -6 route show
2001:470:xc:xxx::1 dev he-ipv6 metric 1024
2001:470:xc:xxx::/64 via :: dev he-ipv6 proto kernel metric 256
2001:470:xd:xxx::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth1 proto kernel metric 256
fe80::/64 dev br0 proto kernel metric 256
fe80::/64 dev tap0 proto kernel metric 256
fe80::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth2 proto kernel metric 256
fe80::/64 via :: dev he-ipv6 proto kernel metric 256
default via 2001:470:xc:xxx::1 dev he-ipv6 metric 1024
default dev he-ipv6 metric 1024
Client configuration:
IP configuration on eth0
eth0 Link encap:Ethernet HWaddr 00:50:56:00:02:00
inet addr:195.184.xxx.xxx Bcast:195.184.xxx.xxx Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe00:200/64 Scope:Link
inet6 addr: 2001:470:xd:xxx::6/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:76170 errors:0 dropped:8 overruns:0 frame:0
TX packets:12191 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:44070379 (44.0 MB) TX bytes:2137837 (2.1 MB)
Interrupt:19 Base address:0x14a4
/etc/network/interfaces
iface eth0 inet6 static
address 2001:470:xd:xxx::6
netmask 64
gateway 2001:470:xd:xxx::1
dns-nameservers 2001:470:xd:xxx::5 2001:470:xd:xxx::6
no sysctl parameters.
iptables output
# Generated by iptables-save v1.4.12 on Sun Mar 24 17:37:53 2013
*filter
:INPUT ACCEPT [33599:39575179]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [7271:1209951]
:ssh - [0:0]
:ssh-ddos - [0:0]
-A INPUT -p tcp -m multiport --dports 22 -j ssh-ddos
-A INPUT -p tcp -m multiport --dports 22 -j ssh
-A ssh -s 63.141.228.50/32 -j DROP
-A ssh -j RETURN
-A ssh-ddos -j RETURN
COMMIT
# Completed on Sun Mar 24 17:37:53 2013
ip6tables output:
# Generated by ip6tables-save v1.4.12 on Sun Mar 24 17:39:18 2013
*filter
:INPUT ACCEPT [1142:128841]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2207:193460]
COMMIT
# Completed on Sun Mar 24 17:39:18 2013
ping test from the client:
ping6 -c 4 ipv6.google.com
PING ipv6.google.com(muc03s01-in-x12.1e100.net) 56 data bytes
64 bytes from muc03s01-in-x12.1e100.net: icmp_seq=1 ttl=54 time=63.3 ms
64 bytes from muc03s01-in-x12.1e100.net: icmp_seq=2 ttl=54 time=63.6 ms
64 bytes from muc03s01-in-x12.1e100.net: icmp_seq=3 ttl=54 time=63.0 ms
64 bytes from muc03s01-in-x12.1e100.net: icmp_seq=4 ttl=54 time=62.5 ms
--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3008ms
rtt min/avg/max/mdev = 62.552/63.142/63.602/0.390 ms
traceroute from the client:
traceroute6 ipv6.google.com
traceroute to ipv6.l.google.com (2a00:1450:4016:800::1012) from 2001:470:xd:xxx::6, 30 hops max, 16 byte packets
1 xxxx.xxxxxxx.hu (2001:470:xd:xxx::5) 0.56 ms 0.277 ms 0.171 ms
2 xxxx.tunnel.tserv26.ber1.ipv6.he.net (2001:470:xc:xxx::1) 43.068 ms 42.747 ms 42.109 ms
3 gige-g2-20.core1.ber1.he.net (2001:470:0:220::1) 41.343 ms 40.818 ms 41.344 ms
4 2001:7f8:19:1::3b41:1 (2001:7f8:19:1::3b41:1) 65.077 ms 61.184 ms 59.244 ms
5 2001:4860::1:0:60d (2001:4860::1:0:60d) 59.202 ms 58.63 ms 60.596 ms
6 2001:4860::8:0:3098 (2001:4860::8:0:3098) 59.333 ms 60.176 ms 59.824 ms
7 2001:4860::1:0:336c (2001:4860::1:0:336c) 65.093 ms 62.863 ms 62.805 ms
8 2001:4860:0:1::535 (2001:4860:0:1::535) 62.896 ms 64.405 ms 62.897 ms
9 2a00:1450:8000:1e::1 (2a00:1450:8000:1e::1) 68.002 ms 64.734 ms 64.679 ms
wget test from the client:
wget -O/dev/null http://ipv6.google.com/
--2013-03-24 17:44:26-- http://ipv6.google.com/
Resolving ipv6.google.com (ipv6.google.com)... 2a00:1450:4016:800::1012
Connecting to ipv6.google.com (ipv6.google.com)|2a00:1450:4016:800::1012|:80... failed: Connection timed out.
Retrying.
IPv6 routing from the client:
ip -6 route show
2001:470:xd:xxx::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth1 proto kernel metric 256
fe80::/64 dev eth2 proto kernel metric 256
default via 2001:470:xd:xxx::1 dev eth0 metric 1024
The router and client IPv4 address are different, but all on the same (physical) router. As I see the router not forward any packets to the clients. I already tested the traceroute via http://www.subnetonline.com/pages/ipv6-network-tools/online-ipv6-traceroute.php. BOTH IPv6 was okay.
- The IPv4 and IPv6 addresses has been removed after the problem solved!! -
I couldn't spot any mistakes in your configuration. Running the tunnel and the IPv6 LAN on the same interface stroke me as unusual. But I believe that is correct in your setup. Besides, if that was the source of the problem, the symptoms would look different.
I tried a few variations of traceroute commands towards your addresses, which mostly confirmed what you already observed. But then I got a clue from using traceroute6 with TCP packets.
A telnet command from my computer to 2001:470:6d:318::6 port 80 times out. But a traceroute6 commands with TCP packets send to the exact same port showed success.
Why would two tools testing the exact same thing disagree like that?
I fired up tcpdump and I could see my telnet command causing my kernel to send TCP SYN packets, which 2001:470:6d:318::6 replied to with TCP RST packets. The kernel on my computer just ignored the TCP RST packets and kept retransmitting the SYN packet.
Why would the kernel ignore a TCP RST packet? The first thing that came to my mind was possibly an invalid TCP checksum. So I fired up wireshark to look more closely on the TCP RST packets from your host. Lo and behold, the TCP RST packets did indeed have invalid checksums. I saw checksum B1DD on a couple of TCP RST packets, that should have had checksums 0300 and 64A0.
If the packets had correct checksums when they were sent and got corrupted in transit, it would be unlikely for them to show up at my end with identical checksums. Which leads me to believe they had invalid checksum when they left 2001:470:6d:318::6 in the first place.
So first thing you need to check is if 2001:470:6d:318::6 is running a buggy kernel with a faulty TCPv6 stack. Can you open TCP connections between 2001:470:6d:318::6 and 2001:470:6d:318::1? Have you tried running ssh over IPv6 between them? And are those two hosts running the same kernel version?
If the faulty TCP checksum is not caused by the kernel version, it might have to do with TCP checksum offloading. Some network interfaces can do checksumming in hardware, so the TCP stack just hands the packet to the network interface with an invalid checksum and tells the network interface to do the calculation.
As I understand it, in your case the network interface causing your trouble is a virtual network interface. Could it be that it is emulating a particular piece of hardware with support for TCP checksum offloading on both IPv4 and IPv6, but the emulation of said hardware is incomplete and supports only TCP checksum offloading on IPv4? If that's the case, it means the drive is strictly speaking not compatible with the emulation.
I am just guessing at possible reasons, but if it turns out to really be the case that the problem is caused by kernel using TCP checksum offloading on an emulated network interface, which does not support it, there are three ways it could be solved.
- Upgrade driver to a later version, which may know about this subtle difference between the real network interface and the emulated one.
- Upgrade VMWare to a later version with more complete hardware emulation.
- Just turn off the usage of TCP checksum offloading to let the kernel do it before handing the packet to the network interface.
Looks like you got it working. What did you need to change to get correct checksums on the TCP packets?
Dear kasperd,
Many-many thanks for your help and hints! The problem has been solved as you wrote me the 3rd option. I switched off the ChecksumOffload in the vmware servers. (than reboot the all virtual machines) Everything working normally. Packets are okay and my VPN clients same receive the IPV6 address. So, everything restored back to the normal state.
I am very thankful for your analysis/help and of course your time!!!!
Have a nice day, and many thanks again!!!