Youtube over IPv6 / London Tunnel / Save my hair.

[mhisani]

Greetings. I have a tunnel on one of the London servers and it's fantastic. Working a treat and has done for a long time. I noticed today that I was no longer able to browse youtube. Well I could browse the pages but not the videos. I was met with "an error occured please try later" on all videos. I've tried clearing browser cache etc to default. Even a different machine. I have noticed that if I disable ipv6 youtube works fine over normal ISP ipv6. Do you have any suggestions? I am not a master of ipv6 but learning all the time. Thanks muchly!

[kasperd]

I have noticed that if I disable ipv6 youtube works fine over normal ISP ipv6.

Was that really what you intended to say? The problems are not necessarily caused by your setup or the tunnel server. It could also be a problem on the youtube side. If anybody else using the same tunnel server would test it as well, then their feedback could help narrow down the cause.

If you want us to help debugging this any further, we will need more information to work with. Packet dumps showing which resources are being fetched in the two cases, and where it gets stuck in case of IPv6, would make it much easier.

It would also help if you'd mention your IPv4 and IPv6 addresses, such that we can probe your connection to see if there are any obvious problems with it.

Finally you can try to use a few of the IPv6 connectivity tests available on the net. I have one on http://netiter.dk/test-ipv6 and Jason Fesler has one on http://test-ipv6.com/

Update:
I have noticed that many people still read this post, and visit the test link, I mentioned above (which btw is migrating to a new URL). So I figured that as long as I can still edit this post, I can use this section to provide as up to date information on the subject as possible.

Currently symptoms suggest the problem is on the YouTube side. It is not a problem on the client or the tunnel server. But various aspects can affect the frequency of the problems leading to the impression, that it might be a problem with a specific tunnel server. As of now, the best guess about a reason is this posting on page two of the thread:

I'm seeing a similar issue with the 403's.

If you use Chrome's developer tools on the "Network" tab you can watch it download chunks of the video.

The request is to http://o-o---preferred---sn-aigezn7y---v3---lscache6.c.youtube.com/videoplayback
and then a whole bunch of query parameters. Part of that parameter is the IP address... I can see my IPv6 address in there no problem.

What I think is happening is YouTube's security is throwing a wobbly, because Chrome is varying between requesting bits from IPv4 and IPv6. The reason for this is one I came up against recently on the customer portal of a DNS provider. Basically, whenever a request is made both an IPv4 and an IPv6 request are made for the same thing simultaneously and whoever sends the ACK back first, is the "fastest" so that connection is used. In the case of very well connected places (e.g. Youtube) the different between v4 and v6 is so negligable, a few milliseconds means one is faster, then the other. This means YouTube see some of the requests from v4, and some from v6, which triggers their protection.

Very annoying and AFAIK nothing you can do about it except not using IPv6.

[kasperd]

I found a couple of IP addresses and tried traceroute. Nothing obviously wrong.

Code Select Expand

traceroute to mhisani-1.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:1d83::1), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.366 ms  0.835 ms  15.189 ms
2  2001:470:27:940::1  62.462 ms  66.677 ms  56.864 ms
3  2001:470:0:11e::1  60.647 ms  21.930 ms  32.782 ms
4  2001:470:0:22f::1  58.123 ms  62.335 ms  77.702 ms
5  2001:470:0:3f::1  73.057 ms  70.236 ms  85.968 ms
6  2001:470:1f08:1d83::1  65.719 ms  65.879 ms  55.445 ms

Code Select Expand

traceroute to 2001:470:1f08:1d83::2 (2001:470:1f08:1d83::2), 30 hops max, 80 byte packets
1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.367 ms  0.848 ms  6.814 ms
2  2001:470:27:940::1  67.239 ms  76.227 ms  70.950 ms
3  2001:470:0:11e::1  79.207 ms  47.050 ms  58.941 ms
4  2001:470:0:22f::1  76.449 ms  80.736 ms  84.112 ms
5  2001:470:0:217::2  91.184 ms  83.449 ms  84.022 ms
6  2001:470:0:67::2  73.255 ms  73.940 ms  69.212 ms
7  2001:470:1f08:1d83::2  89.540 ms  83.803 ms  87.994 ms

[mhisani]

Well you seem to have got my address and sure enough ipv6 browsing etc works just fine.  Both those tests you linked show me as 10/10 or all green and You are a winner! :)

Is there anything else you'd like from me? Apologies for not being quite the pros you guys are.

I guess it is possible it's youtube.

[kasperd]

You have a problem with fragmented packets. If I do a traceroute to 2001:470:1f08:1d83::1 (which is the server side of the tunnel) using fragmented packets. It works just fine. If I do a traceroute to 2001:470:1f08:1d83::2 (which is the client side of the tunnel) using fragmented packets, I get no reply.

I don't know if that may be the reason for your problem. I think YouTube would use an MSS value low enough to avoid any such MTU issues. Your connection appears to handle 1472 byte packets just fine as long as they are not fragmented.

[kasperd]

Both those tests you linked show me as 10/10 or all green and You are a winner! :)

I checked the log on my test to see which IPv6 address you accessed it from, I then tried to run a traceroute6 to that IP. I got no route to host errors from 2001:470:1f08:1d83::2 (which means they are send by your tunnel endpoint). The errors are ICMPv6 type 1 code 3.

Code Select Expand

1  2001:470:28:940:5d75:c1f4:e0a0:f8ec  0.510 ms  21.087 ms  21.738 ms
2  2001:470:27:940::1  69.427 ms  49.155 ms  55.285 ms
3  2001:470:0:11e::1  55.538 ms  40.181 ms  44.395 ms
4  2001:470:0:22f::1  72.385 ms  79.886 ms  82.161 ms
5  2001:470:0:3f::1  82.639 ms  82.828 ms  83.239 ms
6  2001:470:0:67::2  70.347 ms  74.742 ms  75.096 ms
7  2001:470:1f08:1d83::2  99.331 ms  100.719 ms  104.946 ms
8  *  *  *
9  *  *  *
10  *  *  *
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  2001:470:1f08:1d83::2  479.838 ms !H  352.091 ms !H

I assume that means either the computer you accessed the test from is disconnected from the network right now, or it is using privacy extension and switched to a different IP address since then. Either way, I couldn't test to see if pinging that host behind your gateway would reveal more than just pinging your gateway.

[mhisani]

Apologies I had to disable it on my pc due to needing to look on youtube.

First of all thankyou for being so helpful. I have renabled it on the pc

IPv6 Address. . . . . . . . . . . : 2001:470:1f09:1d83:ed5a:e61c:37a0:5720

Is what my pc is now. I will leave ipv6 on from now on.

Thankyou

[kasperd]

I think YouTube would use an MSS value low enough to avoid any such MTU issues. Your connection appears to handle 1472 byte packets just fine as long as they are not fragmented.

I tried to do a tcpdump while playing a video on YouTube. After being confused by the extremely large number of TCP connections involved in watching just a single video, I took a look on the MSS values.

I found that they would send three different MSS values in their SYN packets: 1410, 1416, and 1440. Subtracting 40 bytes of IPv6 header and 20 bytes of TCP header from your 1472 MTU gives an MSS value of 1412. That means if your gateway was one of the endpoints, it would use an MTU of 1412. But more importantly, any TCP connection you open on which the MSS is 1412 or less, should be immune to the MTU problem I noticed on your connection.

Comparing that with the observed MSS values I realize that I was probably mistaken when I thought YouTube would avoid those problems. Of the MSS values I observed, only the MSS of 1410 was small enough. The MSS of 1416 and 1440 would be too large for your tunnel, thus they would require PMTU discovery.

If the tunnel is correctly configured on your gateway, then PMTU discovery may still work, and the MSS sent by your gateway will be only 1412. But hosts on your LAN know nothing about the tunnel and presumably use native IPv6 when communicating with the gateway, thus those will use an MSS of 1440.

That means the segment size agreed upon by the endpoints while setting up the TCP connections is going to be too large, and PMTU discovery will be required. Some TCP stacks will not reduce the size of the segment that triggered the packet too big message. So one end of the connection will send a 1440 byte segment, which with TCP and IPv6 headers mean a 1500 byte IPv6 packet. That will trigger a packet too big error from the tunnel endpoint. The sending host could retransmit the segment as two smaller segments, but I have observed systems that instead will fragment the segment.

If the fragmentation issue that I mentioned earlier affects not just packets for the gateway but also the computers that you try to stream videos from, then it could cause TCP connections to stall.

This is all just a hypothesis, we would need to see actual packet dumps from your network to confirm it.

[mhisani]

Okay I won't lie I do not know how to do what you're asking of me. Would you be able to point me in the right direction? Many thanks

[kasperd]

2001:470:1f09:1d83:ed5a:e61c:37a0:5720

That is different from the IP I found in my log. The IP you mention does not respond to ICMP echo requests, which makes debugging harder. I do believe it is the correct IP though as I observer different behaviour when I do a traceroute to that IP compared to other random IPs in your /64.

A traceroute to 2001:470:1f09:1d83:ed5a:e61c:37a0:5720 shows replies all the way until your gateway, then no replies from that point on. Which is most likely caused by the gateway forwarding the echo requests, and your computer ignoring them.

A traceroute to random addresses in your /64 OTOH results in no route to host errors from your gateway. The no route to host errors appear to show up only a few hops after the gateway, but that is probably just caused by the number of probes the traceroute command sends before the neighbour discovery times out on the gateway.

So I believe the difference I observe between the two IPs is due to the gateway actually performing a successful neighbour discovery on one IP and not on the other.

Figuring out what happens with regards to fragmentation is quite difficult when I cannot send a packet to the computer and get a reply. I have ideas for how I might be able to come up with some code changes on my end, that could find out more, but I don't know when I would have time to actually write that code.

If you can make your computer respond to echo request, I can certainly find out more.

It would also be convenient to look at some packet dumps. Do you have wireshark, tcpdump or something similar installed on the computer or the gateway?

[mhisani]

I could use any tools like that admittedly I'd need to know how.

Also I managed to get IPv6 pinging working seems one of the windows firewall rules said reply to ping on local subnet only fixed that at least. hope that helps.

[kasperd]

I could use any tools like that admittedly I'd need to know how.

It would be most useful to run tcpdump on the tunnel endpoint itself and analyse the packet traces with wireshark afterwards. However it is probably easier to get started with those tools by just running wireshark on the computer that you are attempting to connect to YouTube from.

Before you try to access the video, start wireshark. Then choose to start a packet trace on the network interface you will use for the IPv6 communication. Next try to access the video. As soon as you get an error go back to wireshark and stop the capture. If you succeed in getting some actual packets this way, then choose to save the trace to a file.

Let me know if this is doable. Then I'll explain what we can do to analyse the dump.

Also I managed to get IPv6 pinging working seems one of the windows firewall rules said reply to ping on local subnet only fixed that at least. hope that helps.

It still doesn't respond. I can still see it is online as I get different result on that IP compared to others on the segment.

I noticed that your gateway does do something weird with fragmented packets. If I try to do a traceroute with fragmented packets to an unused IP on your network, I get no route to host replies from your gateway. Those no route to host replies contain the first fragment of the echo request packets. That is not surprising. However what is surprising is that I only get those messages back, if I send both fragments. If I only send the first fragment of a ping, then the gateway does not send the error back, even though it should.

It looks as if the gateway is trying to reassemble fragmented packets, which it is not supposed to do. The gateway might contain some sort of firewall that will reassemble fragmented packets in order to decide which filtering rules to apply.

[thermionic]

I'm also seeing the same effect, in that I can quite happily view youtube video content on ipv4, but all video content errors with "An error occurred. Please try again later"

I have a Cisco 1812 (running 15.1) terminating a BT FTTC connection (PPPoE, but with RFC4638 so MTU is 1500) and terminating the HE.net tunnel, then all traffic goes through a Cisco ASA Firewall (running 8.4.4)

As this was working quite happily a few weeks ago (been on holiday...) and looking at my rancid diffs, there have been no changed to either the ASA or the 1812, I'm pointing the finger at YouTube....

[kasperd]

I'm also seeing the same effect, in that I can quite happily view youtube video content on ipv4, but all video content errors with "An error occurred. Please try again later"

You are also using tserv5.lon1. It could be that the YouTube streaming servers serving your area are to blame. I did some traceroutes towards your network, and I did not observe any MTU problems. And I could indeed transfer 1480 byte IPv6 packets in both directions.

Could either of you try to look up the AAAA record for mydnsv6.kasperd.net such that I can find out which DNS server you are using?

I do occasionally see the "An error occurred. Please try again later" error, but it always go away if I reload the page.

[mhisani]

mhisani@joggler:~$ host mydnsv6.kasperd.net
mydnsv6.kasperd.net has IPv6 address 2001:470:0:67::2