FreeBSD 10.x: Tunnel Keeps Dropping

[jasonvp]

Hey folks -

I've been using my HE tunnel with a Linux router for a few years, quite successfully.  Recently I decided to move the router to FreeBSD, and have been having no end of difficulty with it and HE.  The tunnel interface gif0 comes up, but I can't always pass packets through it.  Randomly I can, other times I can't.  There doesn't seem to be any pattern with it.

Code Select Expand


# ifconfig gif0
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
tunnel inet [REDACTED] --> 216.66.22.2
inet6 2001:470:7:9af::2 --> 2001:470:7:9af::1 prefixlen 128
inet6 fe80::230:18ff:fea3:b4f8%gif0 prefixlen 64 scopeid 0x6
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

# ping 216.66.22.2
PING 216.66.22.2 (216.66.22.2): 56 data bytes
64 bytes from 216.66.22.2: icmp_seq=0 ttl=60 time=5.184 ms
64 bytes from 216.66.22.2: icmp_seq=1 ttl=60 time=3.025 ms
64 bytes from 216.66.22.2: icmp_seq=2 ttl=60 time=3.906 ms
64 bytes from 216.66.22.2: icmp_seq=3 ttl=60 time=4.440 ms
^C
--- 216.66.22.2 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.025/4.139/5.184/0.787 ms

# ping6 2001:470:7:9af::1
PING6(56=40+8+8 bytes) 2001:470:7:9af::2 --> 2001:470:7:9af::1
^C
--- 2001:470:7:9af::1 ping6 statistics ---
7 packets transmitted, 0 packets received, 100.0% packet loss

The lines in my /etc/rc.conf file:

Code Select Expand


cloned_interfaces="gif0"

# Set up tunnel with HE for IPv6
ifconfig_gif0="tunnel [REDACTED] 216.66.22.2"
ifconfig_gif0_ipv6="inet6 2001:470:7:9af::2 2001:470:7:9af::1 prefixlen 128"

# Routes
ipv6_defaultrouter="2001:470:7:9af::1"


Both PF and IPFW are running.  IPFW is running in "wide open" mode; it's only being used to block incoming port 25 connections from various /32s.  PF is the main filter for my entire network, and I'm logging all drops to pflog0.  When I tcpdump that interface, I see nothing being dropped by PF when I do a ping6 to the HE side of the tunnel.

It seems like the server in ASH isn't responding to my pings for some reason.  Watching tcpdump on the gif0 interface during a ping6:

Code Select Expand


# tcpdump -i gif0 host jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net
tcpdump: WARNING: gif0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 65535 bytes



07:52:38.484488 IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 89, length 16
07:52:39.548003 IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 90, length 16
07:52:40.590176 IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 91, length 16
07:52:41.594424 IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 92, length 16
07:52:42.601488 IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 93, length 16
07:52:43.640479 IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 94, length 16
07:52:44.679932 IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 95, length 16
07:52:45.688501 IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 96, length 16


You can see my router sending the ICMP echo requests out, but the replies never come back.

Watching on the external interface on my router for the corresponding IPv4 packets shows the same thing:

Code Select Expand


# tcpdump -i bridge0 host tserv2.ash1.he.net
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), capture size 65535 bytes


07:53:48.570579 IP lateapex-gw-br0.lateapex.net > tserv2.ash1.he.net: IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 157, length 16
07:53:49.634078 IP lateapex-gw-br0.lateapex.net > tserv2.ash1.he.net: IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 158, length 16
07:53:50.691303 IP lateapex-gw-br0.lateapex.net > tserv2.ash1.he.net: IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 159, length 16
07:53:51.751552 IP lateapex-gw-br0.lateapex.net > tserv2.ash1.he.net: IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 160, length 16
07:53:52.783320 IP lateapex-gw-br0.lateapex.net > tserv2.ash1.he.net: IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 161, length 16
07:53:53.846814 IP lateapex-gw-br0.lateapex.net > tserv2.ash1.he.net: IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 162, length 16
07:53:54.864553 IP lateapex-gw-br0.lateapex.net > tserv2.ash1.he.net: IP6 jasonvp-1-pt.tunnel.tserv13.ash1.ipv6.he.net > jasonvp-1.tunnel.tserv13.ash1.ipv6.he.net: ICMP6, echo request, seq 163, length 16

Echo requests out, no echo replies back.

I'm still not sure I have everything configured properly.  I don't want to blame HE for it (just yet.. :-)) but I'm not sure what else I can check and/or do.  Any suggestions?

Thanks!

[cholzhauer]

I know it's not a BSD issue as mine's been working for over a year on 10.x

Odds are, it's not a HE problem either.

Stop your firewall and see if you continue to have issues; I suspect your issue is here.

[jasonvp]

Stop your firewall and see if you continue to have issues; I suspect your issue is here.

Well, watching the pflog logging facility showed no hits.  Watching the tcpdump for ping traffic on the interface also showed no hits.  And tcpdump will see the packets on the interface before pf drops them. ;)

Just to humor folks, I briefly dropped it, tried the ping, and it still fails.

[broquea]

And you've made certain your IPv4 endpoint is up to date?

[jasonvp]

And you've made certain your IPv4 endpoint is up to date?

Up to date as in how?  As in:

Code Select Expand


# uname -a
FreeBSD lateapex-gw.lateapex.net 10.2-RELEASE-p7 FreeBSD 10.2-RELEASE-p7

that?  Yes.

The problem may have something to do with FreeBSD's bridging implementation.  I'm on Verizon business class, and they're a bit retarded with the way they allocate networking for their buzi customers.  Customers are put on a shared L3 broadcast domain (ie, a /24) with groups of static IPs allocated out of that /24 (but those groups don't fall within CIDR blocks, naturally).  Up until today, I've been bridging the external side of my router and the side with my public servers, putting an IP on the bridge0 interface, and using PF (iptables in Linux, prior) to filter.

Because the bridge interface is on the same L3 domain as my servers, I had one of my allocated IPv6 addresses on it.  But, it's the same interface as the IPv4 tunnel endpoint.

My gut tells me that that was confusing FreeBSD a bit.  Linux just works with it (Linux's bridging code is vastly more mature, it seems).

I'm now doing something even more stupid: proxy arp.  The interface on my router facing VZ now has a real IP on it, and is acting as the tunnel endpoint.  The interface facing my public servers as a completely phony IPv4 address on it, a real IPv6 address on it, and is proxy arping between the two.  So far, I can get to public IPv6 sites such as GOOG, etc:

Code Select Expand


joker$ telnet -6 www.google.com 80
Trying 2607:f8b0:4009:808::2004...
Connected to www.google.com.
Escape character is '^]'.
get /
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=UTF-8
Content-Length: 1555
Date: Fri, 04 Dec 2015 20:45:34 GMT
Server: GFE/2.0

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 400 (Bad Request)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>400.</b> <ins>That's an error.</ins>
  <p>Your client has issued a malformed or illegal request.  <ins>That's all we know.</ins>
Connection closed by foreign host.

Who'dathunk: using a broken network technology to fix a broken network design.  Go Verizon!

[broquea]

Uh...uname -a is your kernel, I asked about your IPv4 endpoint/address being up to date in the broker. But it looks like you figured it out.

[jasonvp]

Uh...uname -a is your kernel, I asked about your IPv4 endpoint/address being up to date in the broker. But it looks like you figured it out.

Oh, sorry.  I thought you meant the OS on the endpoint being updated.  I completely misunderstood your question.