Possible Routing Problem in HE Network

[steve1515]

Hi. I've discovered that I can't reach "mediaserver-ch1-t1-1-v4v6.pandora.com" which resolves to 2620:106:e002:f00f::21 from my HE tunnel. Here's the output of a trace route:

Code Select Expand

C:\Users\Steve>tracert mediaserver-ch1-t1-1-v4v6.pandora.com

Tracing route to mediaserver-ch1-t1-1-v4v6.pandora.com [2620:106:e002:f00f::21]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  gw.xxxxxxx.com [2001:470:xxxxxxxx]
  2    22 ms    24 ms    22 ms  steve1515-1.tunnel.tserv4.nyc4.ipv6.he.net [2001
:470:1f06:593::1]
  3    17 ms    16 ms    20 ms  ge3-8.core1.nyc4.he.net [2001:470:0:5d::1]
  4    38 ms    34 ms    39 ms  100ge7-2.core1.chi1.he.net [2001:470:0:298::1]
  5    49 ms    51 ms    52 ms  10ge5-7.core1.mci3.he.net [2001:470:0:270::2]
  6    67 ms    71 ms    58 ms  10ge15-4.core1.den1.he.net [2001:470:0:240::1]
  7    86 ms    84 ms    90 ms  10ge13-5.core1.sjc2.he.net [2001:470:0:1b4::1]
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
10     *        *        *     Request timed out.
11     *        *        *     Request timed out.
12     *        *        *     Request timed out.
13     *        *        *     Request timed out.
14     *        *        *     Request timed out.
15     *        *        *     Request timed out.

I discovered this because I can't play Pandora music from Pandora.com when my IPv6 tunnel is active. I did a little digging to find that it appears that the trace stops in HE's network.

Can someone else please confirm that they see this too? (I'm tunneled to the NY server.)

Thanks,
Steve

[kcochran]

Traceroutes can be tricky to see where something stops, since where it actually stops, you don't get replies.  Hop 8 is the peering handoff to Pandora, so a lack of reply there isn't terribly unexpected.  Everything further is within Pandora's network, and it appears they may be filtering that traffic out.  In any case, I'm able to hit a webserver on that IP:

Code Select Expand

> telnet -6 mediaserver-ch1-t1-1-v4v6.pandora.com 80
Trying 2620:106:e002:f00f::21...
Connected to mediaserver-ch1-t1-1-v4v6.pandora.com.
Escape character is '^]'.


[steve1515]

kcochran,

You are correct. I to can get to port 80 via IPv6. I guess that rules out a routing problem. I'll have to go back to pandora and see what they have to say.

Can anyone please confirm that they have Pandora music working with their HE tunnel enabled? That would give me some more information to work with.

Thanks,
Steve

[kernelpanic1]

Steve,

I have been noticing the same issue the past few weeks.  I thought it was a browser issue on my PC but every browser posed the same result.  Finally thought of my IPv6 tunnel, disabling IPv6 on my desktop NIC resolves the issue.  I am also trying to get in touch with Pandora but their first level support is horrible.  Apparently according to them, IPv6 = iPhone 6....

My traces show the same result as yours.

[steve1515]

Thank you so much for your reply. I don't feel as alone now! :) Are you also on the NY tunnel server?

Here's a what I've narrowed it down to... Basically IPv4 works and with IPv6 you can connect to their servers and send data, but you don't get a response back. It seems to be isolated to HE's network because I can use another IPv6 network and all is fine.

Here's some more info...
I got this from a packet capture: http://cont-1.p-cdn.com/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg
It is album art for a song. The IP resolves to 2620:106:e001:f00f::34 and 208.85.46.34 for me.
So, I tried the following two links...
http://208.85.46.34/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg
http://[2620:106:e001:f00f::34]/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg

The IPv4 link works and the IPv6 link just hangs. I did further tests using telnet and it looks like you can connect to the server and send the HTTP request, but the server never sends back any data at all.

Can you try out those links and see what you get? Can anyone else confirm the same thing?

Is it possible that this is still an HE network issue?

Thanks!

[kernelpanic1]

Steve,

Same results here.  The IPv4 link works but the IPv6 does not.  However, I can telnet to the IPv6 address on port 80 and do a get and I do get a response.

Yes, I am also on the NY tunnel server.

I am working on getting in touch with Pandora but I am also curious what HE's take on this is.  I can't narrow down a specific date when this stopped working, but IPv6 was most certainly working in the past to Pandora.

-Adam

[steve1515]

That's interesting. I can see the same thing. It looks like if I request just the root  "GET / HTTP 1.1" I do get a response, but when I request the image, I never get any data back. It's looking like this is entirely on Pandora's end. I'm still waiting for Pandora to reply to me. I'm not sure how to get my support ticket elevated with them.

[kernelpanic1]

Steve,

I was finally put in touch with Network Engineering at Pandora.  At some point over the last few days my path to Pandora did change (not sure if they made any changes) and the traceroute goes though as below:

Tracing route to mediaserver-sv5-t1-1-v6.pandora.com [2620:106:e002:f00f::21]
over a maximum of 30 hops:

  1     1 ms     8 ms     9 ms  2001:470:8b7b:2::1
  2     1 ms     1 ms     1 ms  2001:470:8b7b:f::1
  3    39 ms    40 ms    39 ms  adamjannetta-1.tunnel.tserv4.nyc4.ipv6.he.net [2
001:470:1f06:d7c::1]
  4    35 ms    42 ms    37 ms  ge3-8.core1.nyc4.he.net [2001:470:0:5d::1]
  5     *        *        *     Request timed out.
  6     *       44 ms     *     he-1-4-0-0-cr02.newyork.ny.ibone.comcast.net [20
01:558:0:f53e::1]
  7    52 ms    54 ms    54 ms  be-10305-cr02.350ecermak.il.ibone.comcast.net [2
001:558:0:f572::2]
  8     *       88 ms    85 ms  be-10617-cr02.denver.co.ibone.comcast.net [2001:
558:0:f56a::2]
  9    86 ms    87 ms    83 ms  hu-0-8-0-0-cr01.denverqwest.co.ibone.comcast.net
[2001:558:0:f652::2]
10     *        *        *     Request timed out.
11   104 ms   107 ms   111 ms  be-10925-cr01.9greatoaks.ca.ibone.comcast.net [2
001:558:0:f5e7::2]
12   108 ms   107 ms   107 ms  he-0-13-0-0-pe03.11greatoaks.ca.ibone.comcast.ne
t [2001:558:0:f8e1::2]
13   105 ms   106 ms   111 ms  as40428-7-c.11greatoaks.ca.ibone.comcast.net [20
01:559::522]
14   107 ms   107 ms   106 ms  mediaserver-sv5-t1-1-v6.pandora.com [2620:106:e0
02:f00f::21]

However, streaming was still not working.  Pandora sent me a test link, similar to the ones you were using that is below:
http://[2620:106:e002:f00f::21]/images/test/OK.test

This worked fine.  They mentioned MTU, which I had overlooked since nothing had changed on my end.  But after reviewing I realized that I was not adjusting my MSS down to compensate for 60 bytes of IPv6 overhead from 1480.  I added ipv6 tcp adjust-mss 1420 to my Cisco router and everything is working fine now. 

I'm curious if this is a similar fix for you?

[steve1515]

Adam,

Thank you!!! That did indeed fix it for me. I already had my MTU set to 1480, but I didn't have anything set for MSS in my pfSense setup. I had assumed that it automatically set it based off of MTU.  :-[
Once I set my MSS to 1420 (Entered 1460 in pfSense text box since it automatically subtracts 40), it started working.

Why wouldn't I see problems on other Internet sites?

I'm not greatly familiar with this scheme when it comes to IPv6, but aren't the routers supposed to pass some kind of ICMP info back to my router to indicate that there is a sizing problem? Does this still indicate that some routers in between me and Pandora are not passing this information along due to misconfiguration?

As a side note... I also noticed that my routing changed similar to yours... I now go though Comcast's IPv6 network. This is strange as Pandora has a direct peer with HE's network. I wonder if they did that just because you and me are troubleshooting with them?

I'll have to pass my results along to Pandora. I'll keep updating this thread with any new info.

[njcrawford]

I've been trying to figure this out for a few weeks now, and just stumbled across this thread today... it never occurred to me that MSS might need to be manually adjusted. Like Steve, I assumed it would be automatically adjusted according the the MTU.

After adjusting the MSS down to 1420 in my router, Pandora streaming works again. Just wanted to say thanks!  :)

[kassniwqds]

Steve,

I was finally put in touch with Network Engineering at Pandora.  casesamAt some point over the last few days my path to Pandora did change (not sure if they made any changes) and the traceroute goes though as below:

Tracing route to mediaserver-sv5-t1-1-v6.pandora.com [2620:106:e002:f00f::21]
over a maximum of 30 hops:

  1     1 ms     8 ms     9 ms  2001:470:8b7b:2::1
  2     1 ms     1 ms     1 ms  2001:470:8b7b:f::1
  3    39 ms    40 ms    39 ms  adamjannetta-1.tunnel.tserv4.nyc4.ipv6.he.net [2
001:470:1f06:d7c::1]
  4    35 ms    42 ms    37 ms  ge3-8.core1.nyc4.he.net [2001:470:0:5d::1]
  5     *        *        *     Request timed out.
  6     *       44 ms     *     he-1-4-0-0-cr02.newyork.ny.ibone.comcast.net [20
01:558:0:f53e::1]
  7    52 ms    54 ms    54 ms  be-10305-cr02.350ecermak.il.ibone.comcast.net [2
001:558:0:f572::2]
  8     *       88 ms    85 ms  be-10617-cr02.denver.co.ibone.comcast.net [2001:
558:0:f56a::2]
  9    86 ms    87 ms    83 ms  hu-0-8-0-0-cr01.denverqwest.co.ibone.comcast.net
[2001:558:0:f652::2]
10     *        *        *     Request timed out.
11   104 ms   107 ms   111 ms  be-10925-cr01.9greatoaks.ca.ibone.comcast.net [2
001:558:0:f5e7::2]
12   108 ms   107 ms   107 ms  he-0-13-0-0-pe03.11greatoaks.ca.ibone.comcast.ne
t [2001:558:0:f8e1::2]
13   105 ms   106 ms   111 ms  as40428-7-c.11greatoaks.ca.ibone.comcast.net [20
01:559::522]
14   107 ms   107 ms   106 ms  mediaserver-sv5-t1-1-v6.pandora.com [2620:106:e0
02:f00f::21]

However, streaming was still not working.  Pandora sent me a test link, similar to the ones you were using that is below:
http://[2620:106:e002:f00f::21]/images/test/OK.test

This worked fine.  They mentioned MTU, which I had overlooked since nothing had changed on my end.  But after reviewing I realized that I was not adjusting my MSS down to compensate for 60 bytes of IPv6 overhead from 1480.  I added ipv6 tcp adjust-mss 1420 to my Cisco router and everything is working fine now.  Samsung Galaxy S6 cover

I'm curious if this is a similar fix for you?

yeah, have you got it?

[kernelpanic1]

In speaking with Pandora engineering, they did turn down their HE IPv6 peer when we originally reported the issue which explains the routing change.  They re-enabled the peer tonight and had me retest and everything is still working fine for me.  If I remove ipv6 tcp adjust-mss 1420 streaming does break again.

Seems like this may have been MSS all along...

[steve1515]

What I don't understand is why some websites/servers worked before and Pandora didn't. How were the other sites working when we had the incorrect MSS value set? Does this have something to do with ICMP not being enabled in network routers somewhere along the path?

[DJX]

I'm running into an issue with Pandora streaming as well.
The test URL mentioned in this thread: http://[2620:106:e002:f00f::21]/images/test/OK.test is fine for me.
I'm running a SonicWALL and I can't adjust the MTU or MSS for anything IPv6.

[steve1515]

Does this link work for you?
http://[2620:106:e001:f00f::34]/images/public/amz/5/2/1/0/4003091900125_500W_500H.jpg

Unfortunately, I'm not familiar with SonicWall, so I don't know how to set MTU/MSS. Maybe someone else can chime in?