Possible Routing Problem

[bimmerdriver]

I'm using an HE tunnel for ipv6, connected through a pfsense router. ipv6-test.com and test-ipv6.com both indicate ipv4 and ipv6 are workign properly. However, over the past few days, I've noticed a few websites that aren't accessible, e.g., mail.yahoo.com and speedtest.xfinity.com. tracert -4 and -6 both work, but I can only access the sites when I disable ipv6 on the client, which is windows. I've also noticed that mobiles connected to the wifi are having the same problem. Changing to a different browser doesn't make any difference either. To eliminate the router, I upgraded to the latest version. It made no difference. To eliminate the tunnel, I tested using a different pfsense router, running a development version of pfsense on another computer, which is configured to use native ipv6. The problem doesn't occur on this configuration, so it's looking like there is a routing problem with the tunnel. Any suggestions what to do with this?

[broquea]

If traces work but HTTP(S) doesn't, that isn't a routing issue, that sounds more like PMTUD. Try tuning your MTU on both sides of the tunnel.

[bimmerdriver]

If traces work but HTTP(S) doesn't, that isn't a routing issue, that sounds more like PMTUD. Try tuning your MTU on both sides of the tunnel.

Thanks for the reply. I posted on the pfsense forum. I'll update this thread when I find a solution.

[bimmerdriver]

I ran the ICSI / UC Berkeley Netalyzer. It reported some dns issues as well as this:

IPv6 Path MTU (?): Warning – Your system can receive fragmented traffic, but can not send fragmented traffic over IPv6. The path between our system and your network has an MTU of 1480 bytes. The bottleneck is at IP address 2001:470:X:Y::2. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.

[broquea]

Well then I guess its time to fiddle with your MTU settings on the tunnel. I'd recommend setting both sides to 1280 and raising it until i breaks again. If you are on PPPOE, then your max would be 1472 instead of 1480.

[bimmerdriver]

Well then I guess its time to fiddle with your MTU settings on the tunnel. I'd recommend setting both sides to 1280 and raising it until i breaks again. If you are on PPPOE, then your max would be 1472 instead of 1480.

I set the tunnel to 1280 but it didn't make any difference. Here is the result according to netalyzer:

Your system can receive fragmented traffic, but can not send fragmented traffic over IPv6. The path between our system and your network has an MTU of 1452 bytes. The bottleneck is at IP address 2001:470:0:9d::2. The path between our system and your network does not appear to handle fragmented IPv6 traffic properly.

Also, I reconfigured the DNS to be exactly the same as my other system for which I have no problem accessing mail.yahoo.com. The system that's not working was using a dns forwarder. I changed it to use the resolver and also removed any non-default settings. Now pfsense is the dns resolver for both ipv4 and ipv6. Here is what netalyzer says about the DNS WRT mail.yahoo.com:

One popular name has a moderate anomaly: we are unable to find a reverse name associated with the IP address provided by your ISP's DNS server, although we expected to find a name. This is most likely due to a slow responding DNS server. If you rerun Netalyzr and see this condition remain, it could be due to a misconfiguration on the part of the domain owner, deliberate blocking using DNS, or your DNS server could be misconfigured or enabling a Man-in-the-Middle attack.

What's strange is that I'm seeing the same message on the other system, except I can access mail.yahoo.com. The differences between the systems are the one that's not working is using the latest release version of pfsense (2.3.2_1) and HE tunnel for ipv6. The other is using the latest development version (2.4) and native ipv6. I have no way to prove whether the problem is with pfsense because I can't use 2.3.2_1 with native ipv6. I'll see if I can set up the development version to use the HE tunnel.

[bimmerdriver]

I tried various MTU settings and it made no difference. I found this mtu path utility (http://www.iea-software.com/products/mtupath.cfm) and ran it a few times using ipv4 and ipv6 to various sites. The mtu to google.com is 1500 and it was 1480 to mail.yahoo.com. That's the default for tunnelbroker and even with the router set to 1480 also, it made no difference. I cannot access mail.yahoo.com through the tunnel, but I can access it natively on my other system. I'm really stumped by this. The only notable difference between the two systems is the tunnel.

[bimmerdriver]

The MTU for the tunnel is currently set to the maximum, 1480. I did some testing using ping -6. I found that the largest size for mail.yahoo.com is 1432 and the largest size for google.com is 1452. Does that imply the MSS is 1432 and 1452, respectively? I ran mtupath and it reported 1452 MSS for both, suggesting 1500 "estimated MTU". Not sure why there's a discrepancy. What should I add to the MSS to get the MTU?

Also, I'm trying to set up another pfsense system using 2.4 and the tunnel so I can make a direct comparison between 2.4 native ipv6 and 2.4 tunnel. 2.4 is developmental and has an issue so it might be a while before I can post an update about this.

[bimmerdriver]

I did some additional testing and I'd appreciate some feedback. I can access mail.yahoo.com on my test pfsense system, running 2.4 with native ipv6. I can also access it using a host through a mullvad vpn (dual-stack) terminating in the USA. I cannot access it using my "production" pfsense system, running 2.3.2_1 and with an HE tunnel.

I noticed when I looked at the route table that the route to the server end of the tunnel had mtu of 1280 so I set the mtu of opt1 and the tunnel to 1280. It did not fix the problem. I still can't load mail.yahoo.com. I'm not clear how this route got set to 1280 in the first place, because the MTU of opt1 was originally left at the default.

I also did a ping test to mail.yahoo.com and google.com using the 1280 mtu setting. I can ping -6 both of them up to 1232 bytes.

I've attached the ipv6 routes, opt1 configuration, tracert -6 mail.yahoo.com and the results of the dns and ipv6 sections of the netalyzer test. What's interesting is that now netalyzr is now saying the bottleneck is within the HE network, at 2001:470:0:9d::2. Does that indicate a problem in the HE network? I don't even know if MTU is the cause of this problem or whether it's dns (or something else).

Considering I can access mail.yahoo.com natively and through a vpn, but not through the tunnel, it really looks like the tunnel. What else is there to try? I've been using an HE tunnel to access mail.yahoo.com for several years with no recent changes to my network.

[bimmerdriver]

Any comments on this? I've verified MTU settings and I'm getting consistent results using ping and mtu path. It's really looking like something is messed up in the tunnel.

[cholzhauer]

If you think it's a tunnel issue, email ipv6@he.net

[bimmerdriver]

If you think it's a tunnel issue, email ipv6@he.net

Thanks, I'm in contact with support.

I have a question about the normal / expected behaviour of a tunnel. Should it be possible to ping the client and server ends of the tunnel, as well as the allocated gateway address from a remote system? I can ping all three addresses using the wan interface of the router that is connected to the tunnel, but the only address I can ping from a system on another network is the server address of the tunnel. If I try to tracert the addresses, it completes to the server address, but times out part way into the HE network, stopping at tserv1.lax1.he.net. Is that normal?

If I go to ipv6-test.com, icmp to the host doesn't work and the ping test fails. My firewall has a rule to allow ipv6 icmp, but the rule is not firing, suggesting the pings aren't making it to my router. If I use exactly the same firewall rules on my system with native ipv6, I can see the rule firing when the ping test runs.

[cholzhauer]

What number do you get on ipv6-test.com ?

I get 19/20 because my desktop doesn't have RDNS

[bimmerdriver]

I completely reconfigured my router with a new tunnel. I'm using a closer tunnel server with much lower latency and fewer hops. I get 19/20 on ipv6-test.com. The only deficiency is the hostname of for the ipv6 address. The speed and ping tests also work well. I also carefully verified the mtu settings on all of the routes. They are all at 1480 and when I run mtupath to mail.yahoo.com, it's showing 1480. It's working much better than it was before, with the exception that I still cannot access mail.yahoo.com.

I'm really stumped by this. Until a few weeks ago, this had never happened. There were no changes to the router whatsoever. After the problem started happening, I upgraded the router to the latest version. That made no difference. I also changed the dns from the forwarder to the resolver. Also no difference. Now I completely deleted all of the ipv6 configuration and the tunnel, reconfigured it and it's still not working.

[cholzhauer]

Works OK for me, where is your tunnel from?