Hurricane Electric's IPv6 Tunnel Broker Forums

Tunnelbroker.net Specific Topics => Questions & Answers => Topic started by: Steak on October 29, 2014, 10:00:33 AM

Title: So, the current Akamai IPv6 problem
Post by: Steak on October 29, 2014, 10:00:33 AM
For about a month now, there's been problems connecting to any site hosted by akamai under 2a02:26f0:: over an IPv6 tunnel.

This isn't an HE problem, it seems to affect anyone using a tunnelbroker, for example, see this thread on sixxs:

https://www.sixxs.net/forum/?msg=general-12378937

Unfortunately this includes content servers for a lot of major websites such as Facebook, Twitter, Linkedin, Cisco - meaning any webpage which unconditionally loads javascript from those sites (such as any site with facebook integration loading from connect.facebook.com) will hang and never finish loading (you'd expect the browser to time the connection out, but it doesn't)

I've tried all sorts of MTU tweaks to try and manually hack around the problem until they fix it, but nothing works.

I was wondering if anyone else here had successfully come up with a workaround?
Title: Re: So, the current Akamai IPv6 problem
Post by: cholzhauer on October 30, 2014, 08:20:49 AM
I haven't had any issues nor have my users reported any issues.  What tunnel server are you using?
Title: Re: So, the current Akamai IPv6 problem
Post by: DJX on October 30, 2014, 09:08:22 AM
I can connect to everything but Cisco.com
It has been this way for months.

Tracing route to cisco.com [2001:420:1101:1::a]
over a maximum of 30 hops:

  1     3 ms     1 ms     1 ms  core3750x.djxmmx.net [2001:470:xxxx:xxxx:xxxx:xxxx:xxxx:fffe]
  2    35 ms    34 ms    34 ms  djx-1.tunnel.tserv13.ash1.ipv6.he.net [2001:470:7:a40::1]
  3    29 ms    39 ms    29 ms  ge4-12.core1.ash1.he.net [2001:470:0:90::1]
  4    46 ms    35 ms    38 ms  100ge5-1.core1.nyc4.he.net [2001:470:0:299::2]
  5    37 ms    38 ms    37 ms  as7018-att.10gigabitethernet2-3.core1.nyc4.he.net [2001:470:0:1dd::2]
  6    72 ms    77 ms    73 ms  n54ny22crs.ipv6.att.net [2001:1890:ff:ffff:12:122:130:170]
  7    77 ms    74 ms    71 ms  wswdc22crs.ipv6.att.net [2001:1890:ff:ffff:12:122:3:38]
  8    69 ms    71 ms    71 ms  attga21crs.ipv6.att.net [2001:1890:ff:ffff:12:122:1:173]
  9    74 ms    70 ms    71 ms  dlstx22crs.ipv6.att.net [2001:1890:ff:ffff:12:122:28:174]
10    72 ms    73 ms    86 ms  dlstx405me3.ipv6.att.net [2001:1890:ff:ffff:12:122:119:9]
11    73 ms    73 ms    75 ms  2001:1890:c00:8701::11b7:3f7f
12    77 ms    73 ms    74 ms  rcdn9-cd1-dmzbb-gw1-ten1-1.cisco.com [2001:420:1100:5::1]
13    79 ms    76 ms    76 ms  rcdn9-cd2-dmzdcc-gw2-por1.cisco.com [2001:420:1100:1::1]
14    75 ms    75 ms    78 ms  rcdn9-16b-dcz05n-gw2-por2.cisco.com [2001:420:1100:10e::1]
15    74 ms    74 ms    74 ms  www1.cisco.com [2001:420:1101:1::a]
Title: Re: So, the current Akamai IPv6 problem
Post by: Steak on October 31, 2014, 04:25:21 AM
Quote from: cholzhauer on October 30, 2014, 08:20:49 AM
I haven't had any issues nor have my users reported any issues.  What tunnel server are you using?

tserve5.lon1
Title: Re: So, the current Akamai IPv6 problem
Post by: Steak on October 31, 2014, 05:38:51 AM
Also worth mentioning, the Akamai problem only seems to exist on their London cluster, so it'll only happen if you resolve to that node.
Title: Re: So, the current Akamai IPv6 problem
Post by: therrmann on November 07, 2014, 01:20:03 AM
I am seeing exactly the same problems, and it took me quite a while to find out that it was an MTU issue with akamai.

But today I am also seeing problems with www.google.com, rendering IPv6 almost completely unuseable. After many complaints by coworkers, I have now disabled the tunnel on my router, so that IPv6 works in the LAN and happy eyeballs falls back to IPv4 for internet addresses.

As far as I can tell, this must be a major issue for everybody using IPv6 tunnels, so why are there so little problem reports on the internet about this thing? How can I locate the problem in more detail?

Regards
Thomas
Title: Re: So, the current Akamai IPv6 problem
Post by: lobotiger on November 07, 2014, 06:45:25 AM
Ok so it's not just my tunnel that's experiencing ipv6 related problems then.

Noticed it first thing this morning when I couldn't retrieve my gmail because it's mostly all ipv6.  I rebooted my desktop, firewall and I'm about to turn down the HE tunnel because it's affecting too many services hosted by Google.

Any status updates or ways to get it resolved?

LoboTiger
Title: Re: So, the current Akamai IPv6 problem
Post by: Cabal696 on November 07, 2014, 09:20:08 AM
Same issue here with a lot of Google services today. Obviously not related to the Akamai issue, but a real pain.
Title: Re: So, the current Akamai IPv6 problem
Post by: Goofball on November 07, 2014, 10:32:00 AM
Also seeing all sorts of oddball behavior with Google over IPv6 via HE.net tunnel today. Using tserv15.lax1.ipv6.he.net. About to turn down my tunnel so I can work without having to reload things 15 times.
Title: Re: So, the current Akamai IPv6 problem
Post by: NatSocSiD on November 07, 2014, 11:15:37 AM
I'm having same problem since some days (3-4). I'm on tserv4.nyc4
Have to refresh sometime 10-15 time before google.ca or gmail load
Title: Re: So, the current Akamai IPv6 problem
Post by: broquea on November 07, 2014, 12:26:45 PM
Kind of suspecting its the destinations screwing stuff up. Pretty sure I've seen similar reports on ipv6-ops and nanog mailing lists in the last week or so. If it is them, then won't be much to fix on the broker side :(
Title: Re: So, the current Akamai IPv6 problem
Post by: hawk82 on November 07, 2014, 01:08:38 PM
Also confirming issues with loading pretty much any Google site via my HE.net tunnel, tserv13.ash1.ipv6.he.net. It was working fine last night.

Edit: I turned down the MTU from 1480 to 1470 and that seems to have resolved the issue.
Edit2: Disregard, Google pages loaded quickly for awhile but now crawling or barely loading again.
Title: Re: So, the current Akamai IPv6 problem
Post by: NatSocSiD on November 07, 2014, 02:54:58 PM
Quote from: hawk82 on November 07, 2014, 01:08:38 PM
Also confirming issues with loading pretty much any Google site via my HE.net tunnel, tserv13.ash1.ipv6.he.net. It was working fine last night.

Edit: I turned down the MTU from 1480 to 1470 and that seems to have resolved the issue.
Edit2: Disregard, Google pages loaded quickly for awhile but now crawling or barely loading again.

MTU 1480 MSS 1220 = fix
Title: Re: So, the current Akamai IPv6 problem
Post by: lobotiger on November 07, 2014, 03:10:57 PM
Quote from: SiD69 on November 07, 2014, 02:54:58 PM
Quote from: hawk82 on November 07, 2014, 01:08:38 PM
Also confirming issues with loading pretty much any Google site via my HE.net tunnel, tserv13.ash1.ipv6.he.net. It was working fine last night.

Edit: I turned down the MTU from 1480 to 1470 and that seems to have resolved the issue.
Edit2: Disregard, Google pages loaded quickly for awhile but now crawling or barely loading again.

MTU 1480 MSS 1220 = fix

Confirmed that the 1480MTU and 1220MSS numbers worked for my pfsense firewall.  Is this something that's going to have to be permanent or is there a problem somewhere?

LoboTiger
Title: Re: So, the current Akamai IPv6 problem
Post by: hawk82 on November 07, 2014, 08:12:21 PM
Quote from: lobotiger on November 07, 2014, 03:10:57 PM
Quote from: SiD69 on November 07, 2014, 02:54:58 PM
Quote from: hawk82 on November 07, 2014, 01:08:38 PM
Also confirming issues with loading pretty much any Google site via my HE.net tunnel, tserv13.ash1.ipv6.he.net. It was working fine last night.

Edit: I turned down the MTU from 1480 to 1470 and that seems to have resolved the issue.
Edit2: Disregard, Google pages loaded quickly for awhile but now crawling or barely loading again.

MTU 1480 MSS 1220 = fix

Confirmed that the 1480MTU and 1220MSS numbers worked for my pfsense firewall.  Is this something that's going to have to be permanent or is there a problem somewhere?

LoboTiger
Tried that on my pfSense box, and still no dice. Do I need to reboot it for the change to take effect?
Title: Re: So, the current Akamai IPv6 problem
Post by: doktornotor on November 08, 2014, 03:23:02 AM
Quote from: Goofball on November 07, 2014, 10:32:00 AM
Also seeing all sorts of oddball behavior with Google over IPv6 via HE.net tunnel today. Using tserv15.lax1.ipv6.he.net. About to turn down my tunnel so I can work without having to reload things 15 times.

+1. What's up with this? Google services completely broken since yesterday, 4 different locations, tserv27.prg1.ipv6.he.net, tserv1.bud1.ipv6.he.net

>:( >:( >:(

Quote from: hawk82 on November 07, 2014, 08:12:21 PM
Quote from: lobotiger on November 07, 2014, 03:10:57 PM
Quote from: SiD69 on November 07, 2014, 02:54:58 PM

MTU 1480 MSS 1220 = fix

Confirmed that the 1480MTU and 1220MSS numbers worked for my pfsense firewall.  Is this something that's going to have to be permanent or is there a problem somewhere?

LoboTiger
Tried that on my pfSense box, and still no dice. Do I need to reboot it for the change to take effect?

The only thing that worked here is 1280 MTU / 1220 MSS. (Don't forget to change the MTU at https://www.tunnelbroker.net/ (Advanced) as well.) Obviously, I don't consider this to be a permanent solution.
Title: Re: So, the current Akamai IPv6 problem
Post by: robertpenz on November 08, 2014, 04:19:11 AM
Setting MSS in the syn packets to 1420 works for me. I blogged about it here: http://robert.penz.name/971/google-services-seems-to-be-down-if-youre-accessing-them-via-an-ipv6-tunnel-providers/
Title: Re: So, the current Akamai IPv6 problem
Post by: stblassitude on November 08, 2014, 04:30:53 AM
I'm using a FreeBSD 10-stable router with pf, and clamping the MSS to 1220 seems to have fixed the problem for me:
scrub on gif0 max-mss 1220
After reading through this thread fully, I did notice that the default MTU on gif0 was set to 1280 instead of 1480, and the MTU on tunnelbroker.net was set to 1480. I've since increased the MTU on gif0 to 1480.
Title: Re: So, the current Akamai IPv6 problem
Post by: hawk82 on November 08, 2014, 05:06:33 AM
I changed the MSS setting on VPN connections under the Advanced tab in pfSense, not on the OPT1 interface. Oops. I changed that and for the moment Google pages are loading much better.
Title: Re: So, the current Akamai IPv6 problem
Post by: doktornotor on November 08, 2014, 06:14:17 AM
Quote from: hawk82 on November 08, 2014, 05:06:33 AM
I changed the MSS setting on VPN connections under the Advanced tab in pfSense, not on the OPT1 interface. Oops. I changed that and for the moment Google pages are loading much better.

Also note that the pfSense GUI has the MSS settings totally confusing/wrong (https://redmine.pfsense.org/issues/2129#note-7). If you put 1220 into the MSS settings on the GIF tunnel, you end up with 1180, easy to check via

pfctl -sr | grep mss
scrub on gif0 all max-mss 1180 fragment reassemble


So, for pfSense, the MSS value should be 1260 if you have MTU set to 1280 which actually clamps it to 1220:


pfctl -sr | grep mss
scrub on gif0 all max-mss 1220 fragment reassemble

Title: Re: So, the current Akamai IPv6 problem
Post by: hoppmaep on November 08, 2014, 07:04:43 AM
Quote from: SiD69 on November 07, 2014, 02:54:58 PM
Quote from: hawk82 on November 07, 2014, 01:08:38 PM
Also confirming issues with loading pretty much any Google site via my HE.net tunnel, tserv13.ash1.ipv6.he.net. It was working fine last night.

Edit: I turned down the MTU from 1480 to 1470 and that seems to have resolved the issue.
Edit2: Disregard, Google pages loaded quickly for awhile but now crawling or barely loading again.

MTU 1480 MSS 1220 = fix

Google sites were unresponsive for me since yesterday, this fixes it. I wonder what happened on their end?
Title: Re: So, the current Akamai IPv6 problem
Post by: hawk82 on November 08, 2014, 11:54:50 AM
Quote from: doktornotor on November 08, 2014, 06:14:17 AM
Also note that the pfSense GUI has the MSS settings totally confusing/wrong (https://redmine.pfsense.org/issues/2129#note-7). If you put 1220 into the MSS settings on the GIF tunnel, you end up with 1180, easy to check via

pfctl -sr | grep mss
scrub on gif0 all max-mss 1180 fragment reassemble


So, for pfSense, the MSS value should be 1260 if you have MTU set to 1280 which actually clamps it to 1220:


pfctl -sr | grep mss
scrub on gif0 all max-mss 1220 fragment reassemble

Thanks, I confirmed your results and fix.
Title: Re: So, the current Akamai IPv6 problem
Post by: JulioQc on November 08, 2014, 06:11:02 PM
I personally lowered it to 1280 MTU with a slight improvement but some services such as google drive refuse to connect.
Title: Re: So, the current Akamai IPv6 problem
Post by: doktornotor on November 09, 2014, 12:46:29 AM
Quote from: JulioQc on November 08, 2014, 06:11:02 PM
I personally lowered it to 1280 MTU with a slight improvement but some services such as google drive refuse to connect.

You really need the MSS clamping (http://lartc.org/howto/lartc.cookbook.mtu-mss.html), setting MTU is not enough. Regardless, the Google issue should be fixed now:

QuoteDamian Menscher <damian at google.com>
6:44 PM (3 hours ago)

The issue with IPv6 access to Google should now be resolved.  Please let us
know if you're still having problems.
Title: Re: So, the current Akamai IPv6 problem
Post by: JulioQc on November 09, 2014, 06:37:18 PM
Yes its working fine now.

How did you reach out to Google about this anyways?
Title: Re: So, the current Akamai IPv6 problem
Post by: therrmann on November 09, 2014, 10:51:10 PM
I can confirm that MTU 1460 (I am using an additional PPPoE) and MSS 1220 seems to fix at least TCP.

One has to keep in mind that this is a dirty hack that does not help for UDP, ICMP and various other things, and that violates the standards of networking and TCP.

But definitely much better than broken IPv6 or no IPv6.

Regards,
Thomas
Title: Re: So, the current Akamai IPv6 problem
Post by: trevorwarwick on November 10, 2014, 01:23:43 AM
It's going to be interesting to see what happens when Google roll out QUIC across their portfolio - supposed to be happening in the next few months.  Chrome browsers will then prefer to communicate with Google sites over UDP rather than TCP, but so far I can't find any documentation about how they intend to deal with MTU size issues.

I think we may at least expect some teething problems for people running over tunnels that don't provide the end to end 1500 MTU.
Title: Re: So, the current Akamai IPv6 problem
Post by: lobotiger on November 27, 2014, 03:26:42 PM
I think something might still be up.

I've noticed that when accessing G+ via the app on my phone, I don't get the infinite scroll under Everything.  Seems to stop after a short bit.  And then under What's Hot, I notice that the downloading indicator on the page (sideways scrolling colours) keeps going on forever.

Well, I just put back these MTU/MSS values and after restarting the app I'm no longer experiencing the same issues. 

Coincidence?

LoboTiger