Multiple Tunnel interfaces - Only one is working at all times

[Schokobecher]

Hello,

I have the following config:

2 public IPv4 addresses used for 2 HE tunnel (2 accounts) which reside on a VPS (KVM)

These are my two configs for the interfaces:

/etc/network/interfaces.d/he01

Code Select Expand

auto he01
iface he01 inet6 v4tunnel
address 2001:470:1f1c:da5::2
netmask 64
endpoint 216.66.88.98
local 164.132.192.71
ttl 255
up ip -6 rule add from 2001:470:1f1c:da5::2 table he1
up ip -6 route add default via 2001:470:1f1c:da5::1 dev he01 table he1
down ip -6 rule del table he1
down ip -6 route flush table he1



/etc/network/interfaces.d/he02

Code Select Expand

auto he02
iface he02 inet6 v4tunnel
address 2001:470:6c:f4::2
netmask 64
endpoint 216.66.86.114
local 178.33.37.66
ttl 255
up ip -6 rule add from 2001:470:6c:f4::2 dev he02 table he2
up ip -6 route add default via 2001:470:6c:f4::1 table he2
down ip -6 rule del table he2
down ip -6 route flush table he2

I added 2 routing tables for this:
/etc/iproute2/rt_tables

Code Select Expand

#
# reserved values
#
100     he1
101     he2
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep

but now I hit a brick wall.  :-\

Depending on which interface was started first, the second one cant route into the internet. So he01 goes up first, he02 is pingable from the outside, can ping the HE gateway but can't ping ipv6.google.com

Ping via he01:
ping6 ipv6.google.com -I he01

Code Select Expand

PING ipv6.google.com(par10s33-in-x0e.1e100.net (2a00:1450:4007:816::200e)) from 2001:470:1f1c:da5::2 he01: 56 data bytes
64 bytes from par10s33-in-x0e.1e100.net (2a00:1450:4007:816::200e): icmp_seq=1 ttl=55 time=22.5 ms
64 bytes from par10s33-in-x0e.1e100.net (2a00:1450:4007:816::200e): icmp_seq=2 ttl=55 time=22.5 ms

Ping via he02:
ping6 ipv6.google.com -I he02

Code Select Expand

connect: Network is unreachable


Ping gateway of he02:
ping6 2001:470:6c:f4::1 -I he02

Code Select Expand

PING 2001:470:6c:f4::1(2001:470:6c:f4::1) from 2001:470:6c:f4::2 he02: 56 data bytes
64 bytes from 2001:470:6c:f4::1: icmp_seq=1 ttl=64 time=23.1 ms
64 bytes from 2001:470:6c:f4::1: icmp_seq=2 ttl=64 time=23.1 ms


If I ifdown he01, he02 works fine and vice versa. What am I missing?

Here are some diagnostics, if you need anything else let me know  :)

netstat -nr

Code Select Expand

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         164.132.192.1   0.0.0.0         UG        0 0          0 ens3
164.132.192.1   0.0.0.0         255.255.255.255 UH        0 0          0 ens3
178.33.0.0      0.0.0.0         255.255.0.0     U         0 0          0 ens3

ip -6 ro

Code Select Expand

2001:470:6c:f4::/64 dev he02 proto kernel metric 256  pref medium
2001:470:1f1c:da5::/64 dev he01 proto kernel metric 256  pref medium
fe80::/64 dev ens3 proto kernel metric 256  pref medium
fe80::/64 dev he01 proto kernel metric 256  pref medium
fe80::/64 dev he02 proto kernel metric 256  pref medium

route -6n

Code Select Expand

Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
::/0                           2001:470:1f1c:da5::1       UG   1024 1    18 he01
::/0                           ::                         !n   -1  1  1078 lo
::/0                           2001:470:6c:f4::1          UG   1024 0     0 he02
::/0                           ::                         !n   -1  1  1078 lo
2001:470:6c:f4::/64            ::                         Un   256 0     1 he02
2001:470:1f1c:da5::/64         ::                         Un   256 0     1 he01
fe80::/64                      ::                         U    256 0     0 ens3
fe80::/64                      ::                         Un   256 0     0 he01
fe80::/64                      ::                         Un   256 0     0 he02
::/0                           ::                         !n   -1  1  1078 lo
::1/128                        ::                         Un   0   2    82 lo
2001:470:6c:f4::2/128          ::                         Un   0   1     0 lo
2001:470:1f1c:da5::2/128       ::                         Un   0   2     8 lo
fe80::a484:c047/128            ::                         Un   0   1     0 lo
fe80::b221:2542/128            ::                         Un   0   1     0 lo
fe80::f816:3eff:fe26:fb1c/128  ::                         Un   0   1     0 lo
ff00::/8                       ::                         U    256 0     0 ens3
ff00::/8                       ::                         U    256 0     0 he01
ff00::/8                       ::                         U    256 0     0 he02
::/0                           ::                         !n   -1  1  1078 lo


ifconfig he01

Code Select Expand

he01: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1480
        inet6 2001:470:1f1c:da5::2  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::a484:c047  prefixlen 64  scopeid 0x20<link>
        sit  txqueuelen 1  (IPv6-in-IPv4)
        RX packets 117  bytes 221114 (215.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 110  bytes 8327 (8.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ifconfig he02

Code Select Expand

he02: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1480
        inet6 2001:470:6c:f4::2  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::b221:2542  prefixlen 64  scopeid 0x20<link>
        sit  txqueuelen 1  (IPv6-in-IPv4)
        RX packets 2  bytes 208 (208.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2  bytes 208 (208.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


[cholzhauer]

Did this used to work and broke, or is this a new setup?

[Schokobecher]

This a new setup, before I only had only 1 tunnel interface per server

[snarked]

1)  Look carefully at your "up ip -6" rules.  They are not parallel for the two interfaces.  The explicit interface declaration "dev xxx" isn't similar - it hops between the "rule" and "route" subcommands.

2)  If you want packets to go out via both interfaces, you need to do some sort of multi-routing.  This may entail running a routing protocol (BGP, OSPF, etc), or enabling multiple equal path routing in the kernel.  You have multiple default routes, so only the first one found in the routing table will be used in the absence of multi-routing.

[kcochran]

You'll also need to make sure packets from a range of IPs go out their associated tunnel interface.  We do drop spoofed traffic across tunnels, such as would be the case if tunnel1's IPs send traffic out tunnel2.

[divad27182]

1) I suggest avoiding /etc/iproute2/rt_tables and ip rule altogether.  These are for when you want grossly different routing based on some conditions.  On the other hand, you do actually have such a condition configured in this case. but...

2) If you actually want to split things over two tunnels, what you should actually be doing is getting your own AS, two BGP tunnels, and a full routing daemon to manage them.

3) With the rule setup, I think it is not sufficient to specify "-I he02" on ping.  Specify "-I 2001:470:1f1c:da5::2" or "-I 2001:470:6c:f4::2".  The problem is that the source address gets chosen in the table, which is after the rule.

4) Showing "ip -6 ro" isn't enough.  You need "ip -6 rule" and "ip -6 route list table he1", etc...