Youtube over IPv6 / London Tunnel / Save my hair.

[kasperd]

mhisani@joggler:~$ host mydnsv6.kasperd.net
mydnsv6.kasperd.net has IPv6 address 2001:470:0:67::2

I tried switching to that DNS server. But that was not enough to reproduce the problem. Unfortunately that doesn't rule out any possibilities, as users coming through different tunnel servers can still be directed to different streaming servers through other means than DNS.

So, we are probably going to need some packet traces to have any chance of identifying the problem. Has either of you managed to dump some packets while attempting to access a video.

[mhisani]

Youtube seems to be working. Looking into it to be sure.. if so it's annoying to not know the cause :/

[lozzd]

I'm having the same issue, also using a FTTC connection (not with BT) on tserv5.lon1. MTU set to 1280 in advanced options to prevent fragmentation breaking. Everything else is working fine right now.. Just YouTube randomly giving me errors (sometimes in the middle of videos!). I'm not sure if it's intermittently working because occasionally its going via v4 and sometimes via v6 or whether v6 is just being flaky.

[kasperd]

MTU set to 1280 in advanced options to prevent fragmentation breaking.

Just in case somebody isn't aware. There is an MTU setting for each direction on such a tunnel. What you set in the webinterface is the MTU in the direction from the tunnel server to your own gateway. The MTU from your own gateway to the tunnel server is a configuration option on your own gateway.

If you want to avoid MTU problems on the tunnel itself, you can set the MTU in both directions to 1280.

That does however not eliminate all possibilities of MTU problems. What you are avoiding is issues with the IPv4 MTU, and if you go as low as 1280, you at the same time guarantee, that the tunnel is going to be the IPv6 hop with the lowest MTU.

But regardless of which MTU you choose for the tunnel, you are likely going to be in a situation where the MTU of the tunnel is lower than the MTU of the first and last hop of your TCP connections. That means neither end of the TCP connection know in advance, that the PMTU is lower. And thus PMTU is required on IPv6.

So, if you have MTU issues on your IPv4 connection to the tunnel server, you can fix it by tweaking the MTU settings of the tunnel. But MTU issues on the IPv6 connection cannot be solved that way.

Tracking down the root cause of failing PMTU discovery and getting the people responsible for it to fix it, is not always feasible. There is however a workaround, which can eliminate them in case of TCP. Just tweak the MSS value in the TCP SYN packets such that TCP won't be sending too large packets in the first place.

On your own gateway, you can modify the MSS of every SYN packet going through it. If the value is larger than 1220, then it should be reduced to 1220. If the original value is less than 1220, then it is left unmodified. Such a feature exists in ip6tables, and in other software as well.

I'm not sure if it's intermittently working because occasionally its going via v4 and sometimes via v6 or whether v6 is just being flaky.

Yeah, you can really only find out by forcing it to always go over IPv6.

I have experienced situations where I could play videos on devices, which were forced to use IPv6 whenever available. And at the same time I could not play videos on devices, which were configured for IPv4 only.

[quentinsf]

Hi guys -

Just to let you know I'm seeing the same issues (going via tserv5.lon1).  I noticed it when my iPad suddenly stopped playing all embedded YouTube videos, and my laptop gave errors on about 50%.

Have tried reducing the MTU on the tunnel to 1280 as suggested, and my first impression is that it seems to have helped.  Will confirm...

Quentin

[Joelcleanwater]

You should update your flash player maybe it helps.

[dn6vtni89274]

i have encountered the problem/a similar problem at least 2 times
(Youtube with "An error occured please try again later")

- Flash updating didnt help
- sometimes IP4 seemed to work (Firefox about:config -> network.dns.disableIPv6 set to true)
but at the end i switched to a different End Point

After a while i switched back to London and i observed the problem again
- I cleared my youtube+google cookies
- Restarted dnsmasq on my Router with OpenWRT (Flush DNS Caches)
now i think it works again

maybe YT cookies/content protection(unavailability) on video cache servers gets broken with different locations

Edit: Maybe there are some side effects with short IPv6 privacy extensions ?
my radvd got pref. lifetime 1800

[jollino]

I'm having the same problem but I go through tserv1.zrh1.he.net, and this has been happening in the past few weeks alone for me. My gateway is a Raspberry Pi configured via ip (not ifconfig) and nothing changed on my end.

The MTU on the he-ipv6 interface is 1480 (same as on the tunnel server side), but I'm not sure on how to find the MSS, especially not for IPv6. Or are we talking about the 6in4 tunnel? Either way, "route -ee" shows a MSS of 0 for me.

I'm quite confused. :) Thanks for any insight!

[kasperd]

I'm not sure on how to find the MSS, especially not for IPv6. Or are we talking about the 6in4 tunnel? Either way, "route -ee" shows a MSS of 0 for me.

First of all, the IPv4 MSS values do not affect the tunnel. MSS affects only TCP. And since the protocol you are running directly on IPv4 isn't TCP, the MSS setting on the IPv4 routes does not affect your tunnel.

In Ubuntu 10.04 I see quite an inconsistency between the "route" command and the "ip route" command. If I run "route -ee", I see all my IPv4 routes, and the MSS field says 0 for all of them. I am guessing that zero means that the MSS is automatically computed from the MTU. If I run "route -n6ee" I see all my IPv6 routes, but it doesn't show me the MSS value.

It appears the "route" command can only show the MSS for IPv4 routes. On the "ip route" command it is the other way around. If I type "ip route" I see all my IPv4 routes. There is no information about the MSS. If I type "ip -6 route" I see an all my IPv6 routes including the MSS. It does not show me zeros for any of the MSS values. Rather it shows me the actual value regardless of whether it was computed from the MTU or explicitly specified.

However be aware, that you most likely don't care about the MSS settings on your gateway. The MSS settings on your gateway only affects the TCP connections where your gateway is one of the endpoints. When a host on your LAN is communicating through your gateway with another host on the net, the MSS setting on the gateway doesn't affect the communication.

The hack to make MTU problems go away is not a change of the MSS settings on the gateway, rather the trick is to have the gateway modify the MSS on packets going through the gateway. One of the tools which can do this is ip6tables. I haven't tested exactly how ip6tables behaves, so I don't know how elaborate the rules need to be. The documentation gives an example:

Code Select Expand

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtuObviously it should be ip6tables instead of iptables, in order to handle IPv6 packets. I don't know if any other changes are needed to the command line, because it isn't clear from the documentation if --clamp-mss-to-pmtu is intelligent enough to do the right thing, or if you need to compute the MSS yourself.

A different rule, which I think might work better would be:

Code Select Expand

ip6tables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN --mss 1221:65535 -j TCPMSS --set-mss 1220But I haven't tested it.

[jollino]

Thank you for the clarification. I played around with it but I'm at a loss— I admit I don't know the inner details of TCP so I'm going through trial and error.

The ip6tables man page on the Debian-derived distro I have on the Raspberry Pi actually states:

  TCPMSS
      This  target  allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing interface's MTU minus 40 for IPv4 or 60 for IPv6, respectively).  Of course, it can only be used in conjunction with -p tcp.

      This target is used to overcome criminally braindead ISPs or servers which block "ICMP Fragmentation Needed" or "ICMPv6 Packet Too Big" packets.  The symptoms of this problem are that everything works fine from your Linux firewall/router, but machines behind it can never exchange large packets:

      1.  Web browsers connect, then hang with no data received.
      2.  Small mail works fine, but large emails hang.
      3.  ssh works fine, but scp hangs after initial handshaking.

      Workaround: activate this option and add a rule to your firewall configuration like:

              iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

      --set-mss value
             Explicitly sets MSS option to specified value. If the MSS of the packet is already lower than value, it will not be increased (from Linux 2.6.25 onwards) to avoid more problems with hosts relying on a proper MSS.

      --clamp-mss-to-pmtu
             Automatically  clamp  MSS value to (path_MTU - 40 for IPv4; -60 for IPv6).  This may not function as desired where asymmetric routes with differing path MTU exist — the kernel uses the path MTU which it would use to send packets from itself to the source and destination IP addresses. Prior to Linux 2.6.25, only the path MTU to the destination IP address was considered by this option; subsequent kernels also consider the path MTU to the source IP address.

      These options are mutually exclusive.

(Note that it actually says iptables, not ip6tables.)

I have tried both the rule suggested in the manpage with ip6tables, and it gives no error... however it also changes nothing. Forcing it to 1220 bytes via the --set-mss also doesn't seem to work. The rules are properly added, however:

Code Select Expand


Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        
TCPMSS     tcp      anywhere             anywhere             tcpflags: SYN,RST/SYN TCPMSS clamp to PMTU


Code Select Expand


Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        
TCPMSS     tcp      anywhere             anywhere             tcpflags: SYN,RST/SYN TCPMSS set 1220


Also, ip -6 route doesn't show any MSS info for me:

Code Select Expand


pi@raspberrypi ~ $ sudo ip -6 route
::/96 via :: dev sit0  metric 256
2001:470:25:2ad::/64 via :: dev he-ipv6  proto kernel  metric 256
2001:470:25:2ad::/64 dev eth0  metric 1024
2001:470:26:2ad::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 via :: dev he-ipv6  proto kernel  metric 256
ff00::/8 dev eth0  metric 256
ff00::/8 dev he-ipv6  metric 256
default dev he-ipv6  metric 1024


I had even tried rejecting v6 routing to 2a00:1450:4001:c01::/64 (via "ip6tables -A OUTPUT -d 2a00:1450:4001:c01::/64 -j REJECT") just to skip the whole problem, and it seemed to be working... until now.

[kasperd]

The ip6tables man page on the Debian-derived distro I have on the Raspberry Pi actually states

Now that you quote it, I do recall having read that version of the man page before. So looks like you just need to have 2.6.25 or newer in order for the ip6tables rules doing the right thing.

(Note that it actually says iptables, not ip6tables.)

They are similar enough that the author may have felt one man page to cover both was sufficient.

I have tried both the rule suggested in the manpage with ip6tables, and it gives no error...

That's good. If the rules can be added without giving an error, then the mss module must exist for ip6tables.

however it also changes nothing.

What makes you think it changes nothing? In order to verify if it changes anything, you can run tcpdump on both physical interfaces to see if the MSS value is decreased on packets passing through the gateway. Notice that the default packet size limit in tcpdump is too small to see the MSS on tunnelled packets. You need to increase it with the -s flag for tcpdump to see anything.

The rules are properly added, however

I think using ip6tables-save is a better way to see the rules. It is supposed to dump all details. The command you used may leave details out.

I had even tried rejecting v6 routing to 2a00:1450:4001:c01::/64 (via "ip6tables -A OUTPUT -d 2a00:1450:4001:c01::/64 -j REJECT")

The default settings for REJECT are not great. It rejects with an ICMP code, which may not be understood. For TCP packets using --reject-with tcp-reset is more likely to give good results.

[jollino]

Well, I tried running tcpdump and the mss on both the he-ipv6 tunnel interface and the eth0 physical interface is indeed clamped to 1220 for outgoing packets. It's 1440 bytes on eth0, and appears to be both 1410 or 1440 bytes on he-ipv6 depending on the source. This happens both when the videos play and when the error occurs. MTU is 1480 so it should be safe.

The question is: how likely is it that it's actually youtube having issues on its ipv6 servers, maybe even at a higher level than TCP?

[kasperd]

Well, I tried running tcpdump and the mss on both the he-ipv6 tunnel interface and the eth0 physical interface is indeed clamped to 1220 for outgoing packets. It's 1440 bytes on eth0, and appears to be both 1410 or 1440 bytes on he-ipv6 depending on the source. This happens both when the videos play and when the error occurs. MTU is 1480 so it should be safe.

It is theoretically possible that some link on the path has a lower MTU than your tunnel. But if the problem persists even when the MSS is being clamped to 1220, then it isn't an MTU problem. After all the MTU issue was just a guess, because that is something that is recurring with tunnels.

I haven't seen any problems persistent enough for me to really find out what was going on. (Or rather, on the occasions where I have seen a persistent problem, it turned out to be the flash plugin that needed to be restarted.)

The question is: how likely is it that it's actually youtube having issues on its ipv6 servers, maybe even at a higher level than TCP?

It is possible. I can't say what the probability of that is. But I mentioned this thread to a person I know, who actually works on YouTube, then we'll see what happens.

[jollino]

Thank you very much for your help and patience. :)
It's not the Flash plugin in my case, because this happens even when using things like ClickToFlash (which replaces YouTube's flash player with an html5 video element for the mp4 file), and it also happens on my father's iPad (using the 'mobile' version of the site.)

I thought that maybe there may be some actual issue on the web side of YouTube because I noticed earlier that Safari's developer console showed that the video file in question "could not be found on the server", though I couldn't manually try and open the video's direct address as it didn't carry the session over.
Unfortunately Safari 6's developer console is a big step backwards from previous versions, and there's no way to see all the files being downloaded as part of a page unless you open the console before loading the page, and even in that case it's fairly convoluted. The old activity window was perfect for this kind of debugging, even if you were not planning to do it before opening a page. Chrome isn't much better. Oh well, if I manage to catch it in the act I'll be sure I report here. :)

EDIT: Talk about perfect timing. Right after posting this, I had a video choke a couple of minutes in (I noticed that YT downloads 240p and 360p videos in chunks, getting the next only when the current one is halfway done or so.) I happened to have the console open, and it tried to GET four insanely long URLs and several of them report HTTP 403, while the others have no specific message in the console but show a red X.

[lozzd]

I'm seeing a similar issue with the 403's.

If you use Chrome's developer tools on the "Network" tab you can watch it download chunks of the video.

The request is to http://o-o---preferred---sn-aigezn7y---v3---lscache6.c.youtube.com/videoplayback
and then a whole bunch of query parameters. Part of that parameter is the IP address... I can see my IPv6 address in there no problem.

What I think is happening is YouTube's security is throwing a wobbly, because Chrome is varying between requesting bits from IPv4 and IPv6. The reason for this is one I came up against recently on the customer portal of a DNS provider. Basically, whenever a request is made both an IPv4 and an IPv6 request are made for the same thing simultaneously and whoever sends the ACK back first, is the "fastest" so that connection is used. In the case of very well connected places (e.g. Youtube) the different between v4 and v6 is so negligable, a few milliseconds means one is faster, then the other. This means YouTube see some of the requests from v4, and some from v6, which triggers their protection.

Very annoying and AFAIK nothing you can do about it except not using IPv6.