Is HE blocking ICMP replies within its own network?

[snarked]

Traceroute

From news.snarked.org [2001:470:d:4::1:4] to 2607:fc18:dead:beef::7
1  * * *
2  * * *
3  * * *
4  * * *
5  100m-0-0.ash.ipv6.he.net (2001:470:0:40::1)  68.449 ms  68.865 ms  69.904 ms
6  ratbert.glorb.com (2607:fc18:dead:beef::7)  84.828 ms  83.223 ms  81.856 ms

Hops 1-4 are also in HE's network (hop 1 being the Los Angeles tunnel server), yet seemed ICMP blocked.  Please unblock (even if it's just for your 2001:470::/32 addresses).

[bombcar]

I see similar things trying to traceroute6 from one of my tunnels to the other. A ping of the endpoint works, however.

[bombcar]

I have:

Code Select Expand

PING ipv6.bombcar.net(2001:470:d:3a6::2) 56 data bytes
64 bytes from 2001:470:d:3a6::2: icmp_seq=1 ttl=62 time=29.4 ms


but I get:

Code Select Expand

traceroute to ipv6.bombcar.net (2001:470:d:3a6::2) from 2001:470:d:3a2:2e0:81ff:fe27:eb23, 30 hops max, 16 byte packets
1  2001:470:d:3a2::3 (2001:470:d:3a2::3)  1.421 ms  1.094 ms  1.124 ms
2  bombcar-1.tunnel.tserv15.lax1.ipv6.he.net (2001:470:c:3a2::1)  16.473 ms  17.604 ms  16.993 ms
3  * * *
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *


[snarked]

RE - reply #3:  Bad example.

You're trying to traceroute to another tunnel user on the same tunnel server, so that traceroute doesn't cross HE's network.  It crosses only the tunnel server, which did respond.  Your target is at #3.

The traceroute I presented crossed the country on HE's network.  Presumedly the missing hops in mine are:
1)  LA Tunnel Server.
2)  LA Router.
3)  Palo Alto or Denver Router.
4)  Ashburn Router.
Then we pick up with #5: The Ashburn BGP Tunnel Server, and #6, the target.

Perhaps the blocking is the initial UDP packet which is used by traceroute6 to "time out" with "hop limit exceeded" (which is what I should be getting back, but aren't).

[bombcar]

Huh. I wonder why the target didn't respond. Perhaps traceroute6 uses a different part of ICMP?

[kriteknetworks]

Traceroute by default uses UDP, not ICMP, in case you weren't aware....

[snarked]

Outbound, it is UDP (33433-33499), but the replies are ICMP errors (hop limit exhausted).  Regardless, it is blocked and therefore doesn't report.  I am asking for that to be changed.

[broquea]

1) routers when either under load or busy with more important packets will defer ICMP replies.

2) seems fine to me:

Code Select Expand

1  gige-g4-6.core1.lax1.he.net  1.065 ms  0.907 ms  0.982 ms
2  10gigabitethernet4-3.core1.nyc4.he.net  61.991 ms  61.433 ms  61.983 ms
3  10g-2-3.core1.ash1.ipv6.he.net  73.973 ms  68.419 ms  71.933 ms
4  100m-0-0.ash.ipv6.he.net  67.974 ms  68.390 ms  69.029 ms
5  ratbert.glorb.com  81.929 ms  81.423 ms  81.980 ms


[snarked]

It's still dropping those entries for me.  I just ran a traceroute6 again.  This time I tried it from my tunnel endpoint address, 2001:470:c:4::2 - no difference.  Still drops the first 4 hops, and displays the Ashburn BGP router and the destination host only (as hops 5-6).

Per the man page, the default wait is only 5 seconds.  It seems as if these packets are being deferred longer than that.  Increasing the wait to 10 didn't help.

Using your looking-glass feature, the intervening routers/hosts do respond with names and rtts:

core1.lax1.he.net> traceroute ipv6 2607:fc18:dead:beef::7
1 65 ms 61 ms 64 ms 10gigabitethernet4-3.core1.nyc4.he.net (2001:470:0:10e::2)
2 81 ms 75 ms 76 ms 10g-2-3.core1.ash1.ipv6.he.net (2001:470:0:36::1)
3 79 ms 75 ms 75 ms 100m-0-0.ash.ipv6.he.net (2001:470:0:40::1)
4 86 ms 81 ms 88 ms ratbert.glorb.com (2607:fc18:dead:beef::7)
              core1.lax2.he.net> traceroute ipv6 2607:fc18:dead:beef::7
1 10 ms <1 ms <1 ms 10gigabitethernet2-1.core1.lax1.he.net (2001:470:0:72::1)
2 61 ms 64 ms 61 ms 10gigabitethernet4-3.core1.nyc4.he.net (2001:470:0:10e::2)
3 67 ms 69 ms 67 ms 10g-2-3.core1.ash1.ipv6.he.net (2001:470:0:36::1)
4 68 ms 68 ms 69 ms 100m-0-0.ash.ipv6.he.net (2001:470:0:40::1)
5 81 ms 81 ms 81 ms ratbert.glorb.com (2607:fc18:dead:beef::7)

My IPv6 firewall indicates no dropped or rejected packets.  Strange?

Here's another one:

From news.snarked.org [2001:470:d:4::1:4] to 2001:470:992a::3e19:561
1  * * *
2  * * *
3  * * *
4  * * *
5  10g-1-1.core1.ams1.ipv6.he.net (2001:470:0:3f::2)  138.315 ms  138.58 ms  203.924 ms
6  10g-1-1.core1.fra1.ipv6.he.net (2001:470:0:47::2)  277.8 ms  242.478 ms  241.916 ms
7  1g-bge0.tserv6.fra1.ipv6.he.net (2001:470:0:69::2)  185.199 ms  171.194 ms  224.82 ms
8  elend.erje.net (2001:470:992a::3e19:561)  148.844 ms  163.919 ms  259.869 ms

Strangely, the first 4 hops are still masked, but hops 5 and onward are exposed.

[broquea]

Well in response to the original question, we're not filtering.

[snarked]

OK, thanks.  I'm still at a loss of why the first 4 hops are disappearing.  I'll experiment some more and see if it's always the first 4 regardless of route, then see what else they have in common....

[bombcar]

And I can hit them from the other side ...

Code Select Expand

linux ~ # traceroute6 2607:fc18:dead:beef::7
traceroute to 2607:fc18:dead:beef::7 (2607:fc18:dead:beef::7) from 2001:470:d:3a2:2e0:81ff:fe27:eb23, 30 hops max, 16 byte packets
1  2001:470:d:3a2::3 (2001:470:d:3a2::3)  1.451 ms  1.157 ms  1.172 ms
2  bombcar-1.tunnel.tserv15.lax1.ipv6.he.net (2001:470:c:3a2::1)  18.31 ms  17.853 ms  17.194 ms
3  gige-g4-6.core1.lax1.he.net (2001:470:0:9d::1)  16.75 ms  18.393 ms  23.88 ms
4  10gigabitethernet4-3.core1.nyc4.he.net (2001:470:0:10e::2)  84.741 ms  78.24 ms  77.088 ms
5  10g-2-3.core1.ash1.ipv6.he.net (2001:470:0:36::1)  83.785 ms  84.595 ms  83.029 ms
6  100m-0-0.ash.ipv6.he.net (2001:470:0:40::1)  84.385 ms  84.589 ms  82.285 ms
7  ratbert.glorb.com (2607:fc18:dead:beef::7)  96.221 ms  97.931 ms  95.313 ms


[snarked]

I've done all kinds of traceroute6s and for all of them, the first 4 hops (and only those 4) are missing.  I have no idea why.  However, as this isn't an HE-specific problem, if they want the thread closed, I am agreeable.  None of my ip6tables firewall rules use any HL count (although I probably should when it comes to route discovery).

[rwg]

If you're using the traceroute from traceroute.sf.net (common in modern Linux distributions), try using it like:

traceroute6 -N 1 -I foo.example.com

By default, it tries to do 16 simultaneous UDP probes.  By throttling it back to 1 probe at a time and using ICMP instead of UDP, your missing hops might re-appear.

[snarked]

The version of traceroute6 for my distribution of Linux doesn't support ICMP (the -I flag).  It's supported only for IPv4.  Therefore, it's always UDP, and I my firewall rule that lets the traceroute UDP range out does increment (and obviously, for hops 5+, it's even replied to).  I've ruled out the firewall as the problem.