traceroute / mtr 'missing' hops

[darkpixel2k]

I setup two tunnels.  One to my home network and one to a co-located linux server I run.

My home connection is a vyatta router which handles all 6in4 tunneling.  Currently only my laptop is the only machine on my home network with an IPv6 address.

Everything seems to be working perfectly on my home network.  Traceroutes work perfectly, even to my server.

aaron@hoth:~$ traceroute6 quark.ubertragen.com
traceroute to quark.ubertragen.com (2001:470:1f0f:1f7::1) from 2001:470:1f05:33b::2, 30 hops max, 16 byte packets
1  2001:470:1f05:33b::1 (2001:470:1f05:33b::1)  1.376 ms  1.376 ms  0.318 ms
2  darkpixel2k-1.tunnel.tserv3.fmt2.ipv6.he.net (2001:470:1f04:33b::1)  36.562 ms  36.385 ms  35.356 ms
3  1g-3-20.core1.fmt2.ipv6.he.net (2001:470:0:45::1)  45.994 ms  35.332 ms  37.82 ms
4  10gigabitethernet1-2.core1.pao1.he.net (2001:470:0:30::2)  34.242 ms  34.139 ms  33.317 ms
5  10gigabitethernet1-1.core1.lax1.he.net (2001:470:0:34::2)  50.89 ms  45.259 ms  44.641 ms
6  10gigabitethernet1-2.core1.dal1.he.net (2001:470:0:3a::2)  82.076 ms  76.355 ms  82.661 ms
7  1g-bge0.tserv8.dal1.ipv6.he.net (2001:470:0:78::2)  79.63 ms  75.29 ms  74.585 ms
8  quark.ubertragen.com.7.f.1.0.f.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa (2001:470:1f0f:1f7::1)  131.438 ms  125.928 ms  128.179 ms
aaron@hoth:~$

(I think I've corrected the bad reverse DNS record in hop 8, just waiting for the cache to clear...)

From my server pinging back, I get a totally different traceroute:
traceroute to hoth.home.darkpixel.com (2001:470:1f05:33b::2) from 2001:470:1f0e:1f7::2, 30 hops max, 16 byte packets
1  * * *
2  * * *
3  * * *
4  * * *
5  * * *
6  * * *
7  darkpixel2k-1-pt.tunnel.tserv3.fmt2.ipv6.he.net (2001:470:1f04:33b::2)  129.375 ms  127.542 ms  125.754 ms
8  2001:470:1f05:33b::2 (2001:470:1f05:33b::2)  128.081 ms  128.299 ms  131.91 ms


I am able to access services on my home machine, so I know data is getting through correctly, but why are the first 6 hops unknown?

[jimb]

The missing hops are either because ICMP Time Exceeded messages are not being sent, or are being blocked somewhere.  That's how traceroute works.  It sends out UDP packets with a TTL (hop count) starting at 1, and incrementing by one each hop.  Every time it hits a router, the router decrements the TTL.  When the TTL hits zero, the router drops the packet sends back a ICMP Time Exceeded.  That's how traceroute learns each hop.  If the router isn't generating time exceeded, or they're being blocked, the hops will be "missing".

It could also mean that the UDP packets which traceroute6 sends are being silently dropped by hops 1-6, but this is less likely since you are getting a response from hops 7-8.  

It's interesting that you're not getting a response from the other side of the tunnel.  That should be the first or second hop (depending on if you're tracing from the tunnel endpoint or the LAN ... appears u are tracing from the endpoint).

[darkpixel2k]

It's interesting that you're not getting a response from the other side of the tunnel.  That should be the first or second hop (depending on if you're tracing from the tunnel endpoint or the LAN ... appears u are tracing from the endpoint).

My home network site of things terminates the tunnel on a machine running Vyatta.

My colo server terminates the tunnel on it's own 'tun1' interface, and I have assigned an IP from my routed /48 to eth0.  Are you saying that could be the cause?

I did more testing.  The home side of things works perfectly.
I can *connect* to everything from my server (ipv6.google.com, my home network, etc...), but I get those strange traceroutes.

[darkpixel2k]

Also, I can ping the IPv4 address of the tunnel server.  If my IPv6 packets are being converted to IPv4 and passed to the tunnel server, wouldn't this indicate a problem with the tunnel server?

[jimb]

Nah.  I'm just noting that differences in where you are tracing from, and noticing that not even the other side of the tunnel is responding.  I've never played with vyatta so no idea there.  It just seems that it's selectively blocking responses or perhaps outgoing.  

My guess is that the firewall is blocking ICMP time exceeded from "unsolicited" sources.  May want to check the drop logs to see if that's the case.  That'd sort of explain why it'd allow time exceeded from the end node but not intervening nodes, although it's odd that the "next to end" is also allowed even though it's a different IPv6.  ???  That part makes no sense, unless it's some bug in the connection tracking of the SPI firewall or something like that.  

Anyway, you may need to add a rule to simply allow that ICMP type in from anywhere, or switch on some sort of "fixup" or "ALG" for traceroutes for it to work.

[darkpixel2k]

Thanks for the help Jim.  I did kept digging, and I believe there's a problem with the settings listed in this post:
http://www.tunnelbroker.net/forums/index.php?topic=18.0

I got things working by tossing the following into my /etc/network/interfaces file
auto he6
iface he6 inet6 v4tunnel
        endpoint        216.218.224.42 #Tunnel Server IP
        local           68.234.2.106 #Primary IPv4 on my server
        ttl             255
        address         2001:470:1f0e:1f7::2  #My client IPv6 address
        netmask         64
        mtu             1480
        up              route -6 add default gw 2001:470:1f0e:1f7::1   # Set my default IPv6 gateway to the tunnel server IPv6 address
        down            route -6 del default gw 2001:470:1f0e:1f7::1

I still get an error when I bring the interface up, but it appears harmless as the interface does come up and everything is working correctly.

root@quark:~# ifup he6
he6: ERROR while getting interface flags: No such device
root@quark:~#

Thanks again for your help.