Hi. I've discovered that I can't reach "mediaserver-ch1-t1-1-v4v6.pandora.com" which resolves to 2620:106:e002:f00f::21 from my HE tunnel. Here's the output of a trace route:
C:\Users\Steve>tracert mediaserver-ch1-t1-1-v4v6.pandora.com
Tracing route to mediaserver-ch1-t1-1-v4v6.pandora.com [2620:106:e002:f00f::21]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms gw.xxxxxxx.com [2001:470:xxxxxxxx]
2 22 ms 24 ms 22 ms steve1515-1.tunnel.tserv4.nyc4.ipv6.he.net [2001
:470:1f06:593::1]
3 17 ms 16 ms 20 ms ge3-8.core1.nyc4.he.net [2001:470:0:5d::1]
4 38 ms 34 ms 39 ms 100ge7-2.core1.chi1.he.net [2001:470:0:298::1]
5 49 ms 51 ms 52 ms 10ge5-7.core1.mci3.he.net [2001:470:0:270::2]
6 67 ms 71 ms 58 ms 10ge15-4.core1.den1.he.net [2001:470:0:240::1]
7 86 ms 84 ms 90 ms 10ge13-5.core1.sjc2.he.net [2001:470:0:1b4::1]
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
I discovered this because I can't play Pandora music from Pandora.com when my IPv6 tunnel is active. I did a little digging to find that it appears that the trace stops in HE's network.
Can someone else please confirm that they see this too? (I'm tunneled to the NY server.)
Thanks,
Steve
Traceroutes can be tricky to see where something stops, since where it actually stops, you don't get replies. Hop 8 is the peering handoff to Pandora, so a lack of reply there isn't terribly unexpected. Everything further is within Pandora's network, and it appears they may be filtering that traffic out. In any case, I'm able to hit a webserver on that IP:
> telnet -6 mediaserver-ch1-t1-1-v4v6.pandora.com 80
Trying 2620:106:e002:f00f::21...
Connected to mediaserver-ch1-t1-1-v4v6.pandora.com.
Escape character is '^]'.
kcochran,
You are correct. I to can get to port 80 via IPv6. I guess that rules out a routing problem. I'll have to go back to pandora and see what they have to say.
Can anyone please confirm that they have Pandora music working with their HE tunnel enabled? That would give me some more information to work with.
Thanks,
Steve
Steve,
I have been noticing the same issue the past few weeks. I thought it was a browser issue on my PC but every browser posed the same result. Finally thought of my IPv6 tunnel, disabling IPv6 on my desktop NIC resolves the issue. I am also trying to get in touch with Pandora but their first level support is horrible. Apparently according to them, IPv6 = iPhone 6....
My traces show the same result as yours.
Thank you so much for your reply. I don't feel as alone now! :) Are you also on the NY tunnel server?
Here's a what I've narrowed it down to... Basically IPv4 works and with IPv6 you can connect to their servers and send data, but you don't get a response back. It seems to be isolated to HE's network because I can use another IPv6 network and all is fine.
Here's some more info...
I got this from a packet capture: http://cont-1.p-cdn.com/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg
It is album art for a song. The IP resolves to 2620:106:e001:f00f::34 and 208.85.46.34 for me.
So, I tried the following two links...
http://208.85.46.34/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg
http://[2620:106:e001:f00f::34]/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg
The IPv4 link works and the IPv6 link just hangs. I did further tests using telnet and it looks like you can connect to the server and send the HTTP request, but the server never sends back any data at all.
Can you try out those links and see what you get? Can anyone else confirm the same thing?
Is it possible that this is still an HE network issue?
Thanks!
Steve,
Same results here. The IPv4 link works but the IPv6 does not. However, I can telnet to the IPv6 address on port 80 and do a get and I do get a response.
Yes, I am also on the NY tunnel server.
I am working on getting in touch with Pandora but I am also curious what HE's take on this is. I can't narrow down a specific date when this stopped working, but IPv6 was most certainly working in the past to Pandora.
-Adam
That's interesting. I can see the same thing. It looks like if I request just the root "GET / HTTP 1.1" I do get a response, but when I request the image, I never get any data back. It's looking like this is entirely on Pandora's end. I'm still waiting for Pandora to reply to me. I'm not sure how to get my support ticket elevated with them.
Steve,
I was finally put in touch with Network Engineering at Pandora. At some point over the last few days my path to Pandora did change (not sure if they made any changes) and the traceroute goes though as below:
Tracing route to mediaserver-sv5-t1-1-v6.pandora.com [2620:106:e002:f00f::21]
over a maximum of 30 hops:
1 1 ms 8 ms 9 ms 2001:470:8b7b:2::1
2 1 ms 1 ms 1 ms 2001:470:8b7b:f::1
3 39 ms 40 ms 39 ms adamjannetta-1.tunnel.tserv4.nyc4.ipv6.he.net [2
001:470:1f06:d7c::1]
4 35 ms 42 ms 37 ms ge3-8.core1.nyc4.he.net [2001:470:0:5d::1]
5 * * * Request timed out.
6 * 44 ms * he-1-4-0-0-cr02.newyork.ny.ibone.comcast.net [20
01:558:0:f53e::1]
7 52 ms 54 ms 54 ms be-10305-cr02.350ecermak.il.ibone.comcast.net [2
001:558:0:f572::2]
8 * 88 ms 85 ms be-10617-cr02.denver.co.ibone.comcast.net [2001:
558:0:f56a::2]
9 86 ms 87 ms 83 ms hu-0-8-0-0-cr01.denverqwest.co.ibone.comcast.net
[2001:558:0:f652::2]
10 * * * Request timed out.
11 104 ms 107 ms 111 ms be-10925-cr01.9greatoaks.ca.ibone.comcast.net [2
001:558:0:f5e7::2]
12 108 ms 107 ms 107 ms he-0-13-0-0-pe03.11greatoaks.ca.ibone.comcast.ne
t [2001:558:0:f8e1::2]
13 105 ms 106 ms 111 ms as40428-7-c.11greatoaks.ca.ibone.comcast.net [20
01:559::522]
14 107 ms 107 ms 106 ms mediaserver-sv5-t1-1-v6.pandora.com [2620:106:e0
02:f00f::21]
However, streaming was still not working. Pandora sent me a test link, similar to the ones you were using that is below:
http://[2620:106:e002:f00f::21]/images/test/OK.test
This worked fine. They mentioned MTU, which I had overlooked since nothing had changed on my end. But after reviewing I realized that I was not adjusting my MSS down to compensate for 60 bytes of IPv6 overhead from 1480. I added ipv6 tcp adjust-mss 1420 to my Cisco router and everything is working fine now.
I'm curious if this is a similar fix for you?
Adam,
Thank you!!! That did indeed fix it for me. I already had my MTU set to 1480, but I didn't have anything set for MSS in my pfSense setup. I had assumed that it automatically set it based off of MTU. :-[
Once I set my MSS to 1420 (Entered 1460 in pfSense text box since it automatically subtracts 40), it started working.
Why wouldn't I see problems on other Internet sites?
I'm not greatly familiar with this scheme when it comes to IPv6, but aren't the routers supposed to pass some kind of ICMP info back to my router to indicate that there is a sizing problem? Does this still indicate that some routers in between me and Pandora are not passing this information along due to misconfiguration?
As a side note... I also noticed that my routing changed similar to yours... I now go though Comcast's IPv6 network. This is strange as Pandora has a direct peer with HE's network. I wonder if they did that just because you and me are troubleshooting with them?
I'll have to pass my results along to Pandora. I'll keep updating this thread with any new info.
I've been trying to figure this out for a few weeks now, and just stumbled across this thread today... it never occurred to me that MSS might need to be manually adjusted. Like Steve, I assumed it would be automatically adjusted according the the MTU.
After adjusting the MSS down to 1420 in my router, Pandora streaming works again. Just wanted to say thanks! :)
Quote from: adamjannetta on October 15, 2015, 06:38:53 PM
Steve,
I was finally put in touch with Network Engineering at Pandora. casesam (http://www.casesam.co.uk/)At some point over the last few days my path to Pandora did change (not sure if they made any changes) and the traceroute goes though as below:
Tracing route to mediaserver-sv5-t1-1-v6.pandora.com [2620:106:e002:f00f::21]
over a maximum of 30 hops:
1 1 ms 8 ms 9 ms 2001:470:8b7b:2::1
2 1 ms 1 ms 1 ms 2001:470:8b7b:f::1
3 39 ms 40 ms 39 ms adamjannetta-1.tunnel.tserv4.nyc4.ipv6.he.net [2
001:470:1f06:d7c::1]
4 35 ms 42 ms 37 ms ge3-8.core1.nyc4.he.net [2001:470:0:5d::1]
5 * * * Request timed out.
6 * 44 ms * he-1-4-0-0-cr02.newyork.ny.ibone.comcast.net [20
01:558:0:f53e::1]
7 52 ms 54 ms 54 ms be-10305-cr02.350ecermak.il.ibone.comcast.net [2
001:558:0:f572::2]
8 * 88 ms 85 ms be-10617-cr02.denver.co.ibone.comcast.net [2001:
558:0:f56a::2]
9 86 ms 87 ms 83 ms hu-0-8-0-0-cr01.denverqwest.co.ibone.comcast.net
[2001:558:0:f652::2]
10 * * * Request timed out.
11 104 ms 107 ms 111 ms be-10925-cr01.9greatoaks.ca.ibone.comcast.net [2
001:558:0:f5e7::2]
12 108 ms 107 ms 107 ms he-0-13-0-0-pe03.11greatoaks.ca.ibone.comcast.ne
t [2001:558:0:f8e1::2]
13 105 ms 106 ms 111 ms as40428-7-c.11greatoaks.ca.ibone.comcast.net [20
01:559::522]
14 107 ms 107 ms 106 ms mediaserver-sv5-t1-1-v6.pandora.com [2620:106:e0
02:f00f::21]
However, streaming was still not working. Pandora sent me a test link, similar to the ones you were using that is below:
http://[2620:106:e002:f00f::21]/images/test/OK.test
This worked fine. They mentioned MTU, which I had overlooked since nothing had changed on my end. But after reviewing I realized that I was not adjusting my MSS down to compensate for 60 bytes of IPv6 overhead from 1480. I added ipv6 tcp adjust-mss 1420 to my Cisco router and everything is working fine now. Samsung Galaxy S6 cover (http://www.casesam.co.uk/category-galaxy-s6-case-cover-15.html)
I'm curious if this is a similar fix for you?
yeah, have you got it?
In speaking with Pandora engineering, they did turn down their HE IPv6 peer when we originally reported the issue which explains the routing change. They re-enabled the peer tonight and had me retest and everything is still working fine for me. If I remove ipv6 tcp adjust-mss 1420 streaming does break again.
Seems like this may have been MSS all along...
What I don't understand is why some websites/servers worked before and Pandora didn't. How were the other sites working when we had the incorrect MSS value set? Does this have something to do with ICMP not being enabled in network routers somewhere along the path?
I'm running into an issue with Pandora streaming as well.
The test URL mentioned in this thread: http://[2620:106:e002:f00f::21]/images/test/OK.test is fine for me.
I'm running a SonicWALL and I can't adjust the MTU or MSS for anything IPv6.
Does this link work for you?
http://[2620:106:e001:f00f::34]/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg
Unfortunately, I'm not familiar with SonicWall, so I don't know how to set MTU/MSS. Maybe someone else can chime in?
Quote from: steve1515 on October 21, 2015, 04:34:01 PM
Does this link work for you?
http://[2620:106:e001:f00f::34]/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg
Unfortunately, I'm not familiar with SonicWall, so I don't know how to set MTU/MSS. Maybe someone else can chime in?
Negative, that link does not work.
AFAIK, SonicWALL only allows you to change the Link MTU advertised for physical interfaces.
Did Pandora say they were going to continue to work on this issue?
You could try and lower the MTU on your WAN interface to see if that forces the packets going thought the tunnel to be lowered as well. I'm interested to see if this would work for you.
I told Pandora everything works with the MSS change, but haven't heard back if they are still looking at anything. I think they are declaring it fixed though.
--Steve
Quote from: steve1515 on October 22, 2015, 12:09:13 PM
You could try and lower the MTU on your WAN interface to see if that forces the packets going thought the tunnel to be lowered as well. I'm interested to see if this would work for you.
I told Pandora everything works with the MSS change, but haven't heard back if they are still looking at anything. I think they are declaring it fixed though.
--Steve
Yeah that doesn't do anything.
If you've ever seen the configuration pages of SonicWALLs, you'd know how frustrating it is how they only let you configure certain aspects.
In the interfaces page alone, you have to select a radio button for managing either IPv4 or IPv6 side of the interfaces.
The IPv6 side WAN interface isn't doing anything so changing the MTU there is useless.
The IPv4 side WAN interface needs to be at 1500 because that's what my ISP gives me.
The IPv6 tunnel interface just has a few boxes for the address objects and that's about it.
For the IPv6 LAN interface, the only MTU you can configure is the Link MTU advertised in the RA, so that's useless.
How did you get to talk to Pandora anyway?
I tried their contact page and I'm getting nothing, no response.
You should be able to use a smaller MTU than 1500 on your IPv4 WAN interface. Try something like 1280 to test it out. Because the tunnel is over IPv4, it should force your packets to be smaller for both IPv4 and IPv6. If that works, you can then start increasing it until it breaks again.
I used the contact page. It was a web form to start and then they contacted me via email. It took a couple of days for them to get back to me. How many days have you waited? Maybe you could try again. Also check your spam folder, I had one support email end up in there. I almost missed it.
--Steve
Quote from: steve1515 on October 22, 2015, 05:26:21 PM
You should be able to use a smaller MTU than 1500 on your IPv4 WAN interface. Try something like 1280 to test it out. Because the tunnel is over IPv4, it should force your packets to be smaller for both IPv4 and IPv6. If that works, you can then start increasing it until it breaks again.
I used the contact page. It was a web form to start and then they contacted me via email. It took a couple of days for them to get back to me. How many days have you waited? Maybe you could try again. Also check your spam folder, I had one support email end up in there. I almost missed it.
--Steve
It's been two days since I submitted the form on their website.
I tried to lower the MTU on the IPv4 WAN interface to 1280 and it didn't help with the test URL:
http://[2620:106:e001:f00f::34]/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg
Still doesn't load.
:-(
I can confirm the same behavior that steve1515 reported in my own home network using a HE IPv6 tunnel. It's interesting that the MSS/MTU adjustment got it working, but what this really tells me is that PMTU discovery is not functional somewhere along this path. End users should not have to play games with MSS/MTU.
Do you know if there's a way to determine where PMTUD breaks?
Quote
Thanks for writing in. Our Network Operations team is aware of the issue with IPv6.
It looks like you might also be aware of the MSS settings. Most of our listeners have had luck with MSS set to 1440 or 1420, which has been recommended by our Network team.
I'm afraid other than this setting, our Network team does not have another fix at the moment. They are currently investigating a longer term solution.
Sorry for any inconvenience. Please let me know if I can help with anything else.
Thanks so much for listening!
Looks like there is nothing I can do since I can't adjust my MSS or MTU for tunnel interfaces on a SonicWALL.
I asked about an ETA...
Quote
I'm afraid we don't have an ETA as of now, but I can assure you our Network team is currently looking into this issue. They hope to have a solution, or at least more troubleshooting to offer, soon.
Sorry again for any trouble. Feel free to drop us another line in if you have questions in the future.