Strange Routing Issue?

[skiingsean]

Hey all I have an odd issue ::  I have two boxes on my side behind my tunnel (Tunnel is statically set via my cisco 3620).  Both of these boxes are in the same ipv4 broadcast domain and are NAT'd.


One box can route elsewhere just fine:

[root@lotus ~]# traceroute ipv6.google.com
traceroute to ipv6.google.com (2001:4860:0:1001::68), 30 hops max, 40 byte packets
1  2001:470:1f0f:1ce::2 (2001:470:1f0f:1ce::2)  6.646 ms  9.675 ms  11.034 ms
2  modified-1.tunnel.tserv8.dal1.ipv6.he.net (2001:470:1f0e:1ce::1)  60.675 ms  61.179 ms  61.709 ms
3  gige-g2-14.core1.dal1.he.net (2001:470:0:78::1)  65.240 ms  66.061 ms  66.167 ms
4  10gigabitethernet5-2.core1.ash1.he.net (2001:470:0:3b::2)  100.284 ms  101.135 ms  101.239 ms
5  pr61.iad07.net.google.com (2001:504:0:2:0:1:5169:1)  102.193 ms  102.291 ms  102.750 ms
6  * * *
7  2001:4860:0:1001::68 (2001:4860:0:1001::68)  190.545 ms  189.937 ms  189.270 ms
[root@lotus ~]# ping6 -c1 -q ipv6.google.com
PING ipv6.google.com(2001:4860:0:1001::68) 56 data bytes

--- ipv6.google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 202ms
rtt min/avg/max/mdev = 180.361/180.361/180.361/0.000 ms
[root@lotus ~]#
[root@lotus ~]# ip -6 route
unreachable ::/96 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable ::ffff:0.0.0.0/96 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
2001:470:1f0f:1ce::/64 dev eth0  metric 256  expires 20926990sec mtu 1500 advmss 1440 hoplimit 4294967295
unreachable 2002:a00::/24 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:7f00::/24 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:a9fe::/32 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:ac10::/28 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:c0a8::/32 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:e000::/19 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 3ffe:ffff::/32 dev lo  metric 1024  expires 20926991sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
fe80::/64 dev eth0  metric 256  expires 20926990sec mtu 1500 advmss 1440 hoplimit 4294967295
default via fe80::201:96ff:fe24:dc02 dev eth0  proto kernel  metric 1024  expires 1645sec mtu 1500 advmss 1440 hoplimit 64
default via fe80::211:93ff:fe1b:a9ad dev eth0  proto kernel  metric 1024  expires 1724sec mtu 1500 advmss 1440 hoplimit 64
[root@lotus ~]#




Yet the guy sitting right next to lotus:

[root@sasha ~]# traceroute ipv6.google.com
traceroute to ipv6.google.com (2001:4860:0:1001::68), 30 hops max, 40 byte packets
1  2001:470:1f0f:1ce::1 (2001:470:1f0f:1ce::1)  3.470 ms !N  3.924 ms !N  4.363 ms !N
[root@sasha ~]#
[root@sasha ~]# ping6 -c1 ipv6.google.com
PING ipv6.google.com(2001:4860:0:1001::68) 56 data bytes
From 2001:470:1f0f:1ce::1 icmp_seq=1 Destination unreachable: No route

--- ipv6.google.com ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
[root@sasha ~]# ip -6 route
unreachable ::/96 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable ::ffff:0.0.0.0/96 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
2001:470:1f0f:1ce::/64 dev eth0  proto kernel  metric 256  mtu 1500 advmss 1440 hoplimit 4294967295
unreachable 2002:a00::/24 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:7f00::/24 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:a9fe::/32 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:ac10::/28 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:c0a8::/32 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 2002:e000::/19 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
unreachable 3ffe:ffff::/32 dev lo  metric 1024  expires 21332797sec error -101 mtu 16436 advmss 16376 hoplimit 4294967295
fe80::/64 dev vmnet8  metric 256  expires 21268051sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev vmnet1  metric 256  expires 21268051sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev virbr0  metric 256  expires 21268058sec mtu 1500 advmss 1440 hoplimit 4294967295
fe80::/64 dev eth0  metric 256  expires 21332793sec mtu 1500 advmss 1440 hoplimit 4294967295
default via fe80::211:93ff:fe1b:a9ad dev eth0  proto kernel  metric 1024  expires 1628sec mtu 1500 advmss 1440 hoplimit 64
default via fe80::201:96ff:fe24:dc02 dev eth0  proto kernel  metric 1024  expires 1739sec mtu 1500 advmss 1440 hoplimit 64
[root@sasha ~]#



My Tunnel0
router#sh int Tunnel0
Tunnel0 is up, line protocol is up
  Hardware is Tunnel
  Description: Hurricane Electric IPv6 tunnel
  MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 98.199.191.36, destination 216.218.224.42
  Tunnel protocol/transport IPv6/IP, key disabled, sequencing disabled
  Tunnel TTL 255

interface Tunnel0
description Hurricane Electric IPv6 tunnel
no ip address
ipv6 address 2001:470:1F0E:1CE::2/64
ipv6 enable
tunnel source 98.199.191.36
tunnel destination 216.218.224.42
tunnel mode ipv6ip



Any thoughts :)

Thanks,
Sean

[broquea]

What IPv6 addresses are configured on the machines?
Are you using RADVD/DHCPv6 or statically assigning those addresses?
Are both machines running the same Linux platform? Kernel?

I see the working machine trace through "1f0f:1ce::2" and the other machine attempt through "1f0f:1ce::1". What IP from your routed /64 is configured on the Cisco interface that for your LAN? That should always be the same IP (and generally "::1").

[skiingsean]

All IP's are static minus those that are aquired from my MAC but I just allow those to exist and ignore...



[root@sasha ~]# ip addr |grep 2001
    inet6 2001:470:1f0f:1ce::110/64 scope global
    inet6 2001:470:1f0f:1ce:207:e9ff:fe4e:d436/64 scope global dynamic
[root@sasha ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:07:e9:4e:d4:36 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.110/24 brd 192.168.0.255 scope global eth0
    inet6 2001:470:1f0f:1ce::110/64 scope global
       valid_lft forever preferred_lft forever
    inet6 2001:470:1f0f:1ce:207:e9ff:fe4e:d436/64 scope global dynamic
       valid_lft 2591914sec preferred_lft 604714sec
    inet6 fe80::207:e9ff:fe4e:d436/64 scope link
       valid_lft forever preferred_lft forever
3: vmnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:50:56:c0:00:08 brd ff:ff:ff:ff:ff:ff
    inet 172.16.247.1/24 brd 172.16.247.255 scope global vmnet8
    inet6 fe80::250:56ff:fec0:8/64 scope link
       valid_lft forever preferred_lft forever
4: vmnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:50:56:c0:00:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.40.1/24 brd 192.168.40.255 scope global vmnet1
    inet6 fe80::250:56ff:fec0:1/64 scope link
       valid_lft forever preferred_lft forever
5: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
    inet6 fe80::200:ff:fe00:0/64 scope link
       valid_lft forever preferred_lft forever
7: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0
[root@sasha ~]# ping6 ipv6.google.com
PING ipv6.google.com(2001:4860:0:1001::68) 56 data bytes
From 2001:470:1f0f:1ce::1 icmp_seq=1 Destination unreachable: No route

--- ipv6.google.com ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

[root@sasha ~]#





1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:01:29:24:26:75 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.55/24 brd 192.168.0.255 scope global eth0
    inet6 2001:470:1f0f:1ce:201:29ff:fe24:2675/64 scope global dynamic
       valid_lft 2591867sec preferred_lft 604667sec
    inet6 2001:470:1f0f:1ce::55/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::201:29ff:fe24:2675/64 scope link
       valid_lft forever preferred_lft forever
3: sit0: <NOARP> mtu 1480 qdisc noop state DOWN
    link/sit 0.0.0.0 brd 0.0.0.0
[root@lotus ~]# ping6 ipv6.google.com
PING ipv6.google.com(2001:4860:0:1001::68) 56 data bytes
64 bytes from 2001:4860:0:1001::68: icmp_seq=1 ttl=55 time=186 ms
^C
--- ipv6.google.com ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1026ms
rtt min/avg/max/mdev = 186.138/186.138/186.138/0.000 ms
[root@lotus ~]# ping6 ipv6.google.com
PING ipv6.google.com(2001:4860:0:1001::68) 56 data bytes
64 bytes from 2001:4860:0:1001::68: icmp_seq=1 ttl=55 time=181 ms
64 bytes from 2001:4860:0:1001::68: icmp_seq=2 ttl=55 time=180 ms
64 bytes from 2001:4860:0:1001::68: icmp_seq=3 ttl=55 time=182 ms
^C^C
--- ipv6.google.com ping statistics ---
4 packets transmitted, 3 received, 25% packet loss, time 3024ms
rtt min/avg/max/mdev = 180.587/181.451/182.224/0.671 ms
[root@lotus ~]#
[root@lotus ~]#








my static info from he.net is as follows:


        Server IPv4 address:     216.218.224.42
   Server IPv6 address:    2001:470:1f0e:1ce::1/64
   Client IPv4 address:    98.199.191.36
   Client IPv6 address:    2001:470:1f0e:1ce::2/64
   Routed /48:    Allocate
   Routed /64:    2001:470:1f0f:1ce::/64


and both my ip -6 routes go through  2001:470:1f0f:1ce::/64 dev

my .2 and ::2 's are my local gateway's .. Dunno why I chose .2 vs. .1 but that's how I have it here...

[broquea]

Well address assignment on the interfaces looks fine, but I don't see any pastes of the routing table on either system.

What is the output from netstat -rn -A inet6 on both machines?

Also one appears to be virutalized, or is that the actual server and not something virutalized?

[skiingsean]

Sorry for the late reply.  You are correct that one machine is running VM's however that is the host machine which has a few VE's on it; the environment in question is not virtualized.

The first post showed the routing table via ip -6 route .. However I'll paste a more complete below:  Did you mean -r instead of -m on the netstat?

IPv6 Routing table using netstat, output of ipv6 and ipv4 route table on the working machine (lotus)

[root@lotus ~]# netstat -r -A inet6 ; ip addr |grep 2001 ; route -n
Kernel IPv6 routing table
Destination                                 Next Hop                                Flags Metric Ref    Use Iface
2001:470:1f0f:1ce::/64                      *                                       U     256    2        0 eth0   
fe80::/64                                   *                                       U     256    0        0 eth0   
*/0                                         fe80::201:96ff:fe24:dc02                UGDA  1024   2        0 eth0   
*/0                                         fe80::211:93ff:fe1b:a9ad                UGDA  1024   0        0 eth0   
::1/128                                     *                                       U     0      0        1 lo     
lotus.doomed-knowledge.com/128              *                                       U     0      1        1 lo     
2001:470:1f0f:1ce:201:29ff:fe24:2675/128    *                                       U     0      0        1 lo     
fe80::201:29ff:fe24:2675/128                *                                       U     0      0        1 lo     
ff02::1/128                                 ff02::1                                 UC    0      2        0 eth0   
ff02::1:ff00:55/128                         ff02::1:ff00:55                         UC    0      1        0 eth0   
ff00::/8                                    *                                       U     256    0        0 eth0   
    inet6 2001:470:1f0f:1ce:201:29ff:fe24:2675/64 scope global dynamic
    inet6 2001:470:1f0f:1ce::55/64 scope global
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         192.168.0.2     0.0.0.0         UG    0      0        0 eth0
[root@lotus ~]#
quick show that routing is functional
[root@lotus ~]# ping6 -c1 ipv6.google.com
PING ipv6.google.com(2001:4860:0:2001::68) 56 data bytes
64 bytes from 2001:4860:0:2001::68: icmp_seq=1 ttl=58 time=115 ms

--- ipv6.google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 137ms
rtt min/avg/max/mdev = 115.291/115.291/115.291/0.000 ms
[root@lotus ~]#





Same output on non-functional machine:

[root@sasha ~]# netstat -r -A inet6 ; ip addr |grep 2001 ; route -n
Kernel IPv6 routing table
Destination                                 Next Hop                                Flags Metric Ref    Use Iface
2001:470:1f0f:1ce::/64                      *                                       UA    256    1        0 eth0   
fe80::/64                                   *                                       U     256    0        0 vmnet8 
fe80::/64                                   *                                       U     256    0        0 vmnet1 
fe80::/64                                   *                                       U     256    0        0 virbr0 
fe80::/64                                   *                                       U     256    0        0 eth0   
*/0                                         fe80::211:93ff:fe1b:a9ad                UGDA  1024   4        0 eth0   
*/0                                         fe80::201:96ff:fe24:dc02                UGDA  1024   0        0 eth0   
localhost6.localdomain6/128                 *                                       U     0      0        1 lo     
sasha.doomed-knowledge.com/128              *                                       U     0      1        1 lo     
2001:470:1f0f:1ce:207:e9ff:fe4e:d436/128    *                                       U     0      0        1 lo     
fe80::207:e9ff:fe4e:d436/128                *                                       U     0      0        1 lo     
ff02::1/128                                 ff02::1                                 UC    0      2        0 eth0   
ff02::fb/128                                ff02::fb                                UC    0      26       0 eth0   
ff00::/8                                    *                                       U     256    0        0 vmnet8 
ff00::/8                                    *                                       U     256    0        0 vmnet1 
ff00::/8                                    *                                       U     256    0        0 virbr0 
ff00::/8                                    *                                       U     256    0        0 eth0   
    inet6 2001:470:1f0f:1ce::110/64 scope global
    inet6 2001:470:1f0f:1ce:207:e9ff:fe4e:d436/64 scope global dynamic
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
removed  10.2.130.160    255.255.255.224 UG    0      0        0 cscotun0
172.16.247.0    0.0.0.0         255.255.255.0   U     0      0        0 vmnet8
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
192.168.40.0    0.0.0.0         255.255.255.0   U     0      0        0 vmnet1
172.16.208.0    10.2.130.160    255.255.240.0   UG    0      0        0 cscotun0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
10.0.0.0        10.2.130.160    255.0.0.0       UG    0      0        0 cscotun0
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 cscotun0
0.0.0.0         192.168.0.2     0.0.0.0         UG    0      0        0 eth0
[root@sasha ~]#
still no joy
[root@sasha ~]# ping6 -c1 ipv6.google.com
PING ipv6.google.com(2001:4860:0:2001::68) 56 data bytes
From 2001:470:1f0f:1ce::1 icmp_seq=1 Destination unreachable: No route

--- ipv6.google.com ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

[root@sasha ~]#

I have a few minutes before work I'll reexamine my paste's and compare and see if I can see anything new ..

Thanks again :)

[skiingsean]

Flags    The flags field in the routing table shows the state of the route:

A
    An Active Dead Gateway Detection is enabled on the route



Hmm I'm thinking this might have something to do with it but I can actually ping the router's tunnel endpoint but not the HE endpoint.

[root@sasha ~]# ping6 -c1 2001:470:1f0f:1ce::1
PING 2001:470:1f0f:1ce::1(2001:470:1f0f:1ce::1) 56 data bytes
64 bytes from 2001:470:1f0f:1ce::1: icmp_seq=1 ttl=64 time=2.88 ms

--- 2001:470:1f0f:1ce::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.884/2.884/2.884/0.000 ms
[root@sasha ~]#

[skiingsean]

Just in case anyone asks I can reproduce the same results with my VPN tunnel shutdown and after an '/etc/init.d/network restart'  let me know if you want me to reproduce the results and paste again for show:

Sean

[skiingsean]

And Voila..  I still have no idea why this was not working fundamentally since I could contact the gateway, I could ping it, I could hit the tunnel enpoint (local) and it was that enpoint that replied.. But showing the FLAGS - UA told me that it was an 'active dead gateway' so I added a IPV6_DEFAULTGW to this servers ifcfg init script and it works ..

Thanks all, and ideas or advice or thoughts I'd still welcome .. :)

[broquea]

Bah, here i was writing up this big thing with pastes from my routing table, and examples of how I set my static configurations, and hit preview only to see you just figured out the problem.

Good to hear you solved it :)

But basically I was going to point out that both machines had default routes pointing to the exact same link-local addresses, and to specify the default gw in a config file.

[skiingsean]

Hehehe :) Thanks a million for taking the time .. By chance would you know how "Active Dead Gateway" even functions with IPv6 for all that I've read in this short bit I can only find references to AIX routing and even more from what I've learned ADG only works based on arp .. Since IPv6 doesn't use ARP and rather ND is there just little documentation explaining the ipv6 implimentation of ADG or was this something that should not have been?

Sean