trouble loading some CloudFlare pages through tunnel

[uhoreg]

I'm having problems loading some pages served by CloudFlare through my IPv6 tunnel (via the Toronto server).  The connection seems to die part-way through the HTTP connection, at different times, resulting in a timeout.  Sometimes it dies right at the beginning, and sometimes in the middle.  Pinging the server shows no packet loss.  As far as I can tell, it only happens with CloudFlare (e.g. Google, he.net, and sixxs.net work fine.)  Any suggestions for how to debug this?

This is the output from ifconfig on my router:

Code Select Expand

he-ipv6   Link encap:IPv6-in-IPv4 
          inet6 addr: xxxx::xxxx:xxxx/128 Scope:Link
          inet6 addr: 2001:xxxx:xxxx:xxxx::2/64 Scope:Global
          UP POINTOPOINT RUNNING NOARP  MTU:1280  Metric:1
          RX packets:1138198 errors:0 dropped:0 overruns:0 frame:0
          TX packets:849636 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:809197435 (771.7 MiB)  TX bytes:288186600 (274.8 MiB)


[broquea]

And the destination hosts/sites are? Maybe its more widespread than just an HE tunnel?
Try a tracepath6 to the destination?
Did you set the MTU on the HE side to 1280?

[uhoreg]

There are many different sites that are affected.  My own website included...

Traceroute6 from my local computer to my website gives:

Code Select Expand

$ traceroute6 www.uhoreg.ca
traceroute to www.uhoreg.ca (2400:cb00:2048:1::681c:1655), 30 hops max, 80 byte packets
1  gateway.home.uhoreg.ca (2001:470:1d:1da::1)  3.027 ms  2.984 ms  5.499 ms
2  uhoreg-1.tunnel.tserv21.tor1.ipv6.he.net (2001:470:1c:1da::1)  20.606 ms  23.401 ms  25.916 ms
3  ge2-5.core1.tor1.he.net (2001:470:0:c0::1)  25.920 ms  40.387 ms  40.435 ms
4  100ge13-1.core1.chi1.he.net (2001:470:0:2db::1)  43.732 ms 100ge1-2.core1.nyc4.he.net (2001:470:0:2dc::1)  43.616 ms 100ge13-1.core1.chi1.he.net (2001:470:0:2db::1)  43.672 ms
5  xe-0-0-0.edge01.ewr01.as13335.net (2001:504:f::1:3335:1)  40.308 ms . (2001:504:0:4:0:1:3335:1)  43.608 ms  46.177 ms
6  2400:cb00:11:1024::6ca2:da3e (2400:cb00:11:1024::6ca2:da3e)  49.534 ms 2400:cb00:14:1024::6ca2:d962 (2400:cb00:14:1024::6ca2:d962)  44.992 ms  47.541 ms


Tracepath6 stops giving responses after 5 hops:

Code Select Expand

tracepath6 www.uhoreg.ca
1?: [LOCALHOST]                        0.030ms pmtu 1500
1:  gateway.home.uhoreg.ca                                4.026ms
1:  gateway.home.uhoreg.ca                                3.414ms
2:  gateway.home.uhoreg.ca                                6.547ms pmtu 1280
2:  uhoreg-1.tunnel.tserv21.tor1.ipv6.he.net             44.103ms
3:  ge2-5.core1.tor1.he.net                              34.447ms
4:  100ge1-2.core1.nyc4.he.net                           44.334ms
5:  xe-0-0-0.edge01.ewr01.as13335.net                    48.162ms
6:  no reply
7:  no reply
8:  no reply
...
     Too many hops: pmtu 1280
     Resume: pmtu 1280


Yes, I set the MTU on the HE side to 1280.

Problems seem to have started within the past few months.  It's hard to tell exactly when it started, because most of my browsing is done via tor, but I'm pretty sure that, say, a year ago, everything was working fine.

[namronorman]

I am seeing the same thing as OP. I am using the Ashburn, VA endpoint and am having difficulty seeing Cloudflare destinations all of a sudden.

[Napsterbater]

Just adding that I have the same issue. (oh but this just started today)

Code Select Expand

Non-authoritative answer:
Name:    autoplicity.com
Addresses:  2400:cb00:2048:1::a29f:fb90
          2400:cb00:2048:1::a29f:fa90
          162.159.251.144
          162.159.250.144


C:\WINDOWS\system32>tracert autoplicity.com

Tracing route to autoplicity.com [2400:cb00:2048:1::a29f:fa90]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  car1.****.local [2001:470:****:1::3]
  2    25 ms    23 ms    24 ms  ****-2.tunnel.tserv13.ash1.ipv6.he.net [2001:470:7:****::1]
  3    28 ms    23 ms    23 ms  ge4-12.core1.ash1.he.net [2001:470:0:90::1]
  4    20 ms    20 ms    20 ms  xe-0-1-3.edge01.iad02.as13335.net [2001:504:0:2:0:1:3335:1]
  5    19 ms    20 ms    20 ms  2400:cb00:2048:1::a29f:fa90

Trace complete.

C:\WINDOWS\system32>tracert 2400:cb00:2048:1::a29f:fb90

Tracing route to 2400:cb00:2048:1::a29f:fb90 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  car1.****.local [2001:470:****:1::3]
  2    24 ms    23 ms    23 ms  ****-2.tunnel.tserv13.ash1.ipv6.he.net [2001:470:7:****::1]
  3    22 ms    21 ms    24 ms  ge4-12.core1.ash1.he.net [2001:470:0:90::1]
  4    21 ms    19 ms    20 ms  xe-0-1-3.edge01.iad02.as13335.net [2001:504:0:2:0:1:3335:1]
  5    20 ms    19 ms    20 ms  2400:cb00:2048:1::a29f:fb90

Trace complete.

Non-authoritative answer:
Name:    sourcefed.com
Addresses:  2400:cb00:2048:1::681c:d22
          2400:cb00:2048:1::681c:c22
          104.28.13.34
          104.28.12.34


C:\WINDOWS\system32>tracert 2400:cb00:2048:1::681c:d22

Tracing route to 2400:cb00:2048:1::681c:d22 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  car1.***.local [2001:470:***:1::3]
  2    24 ms    24 ms    24 ms  ***-2.tunnel.tserv13.ash1.ipv6.he.net [2001:470:7:***::1]
  3    22 ms    21 ms    23 ms  ge4-12.core1.ash1.he.net [2001:470:0:90::1]
  4    20 ms    20 ms    19 ms  xe-0-1-3.edge01.iad02.as13335.net [2001:504:0:2:0:1:3335:1]
  5    20 ms    20 ms    20 ms  2400:cb00:2048:1::681c:d22

Trace complete.

C:\WINDOWS\system32>tracert 2400:cb00:2048:1::681c:c22

Tracing route to 2400:cb00:2048:1::681c:c22 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  car1.***.local [2001:470:***:1::3]
  2    25 ms    44 ms    24 ms  ***-2.tunnel.tserv13.ash1.ipv6.he.net [2001:470:7:***::1]
  3    23 ms    19 ms    21 ms  ge4-12.core1.ash1.he.net [2001:470:0:90::1]
  4    20 ms    20 ms    20 ms  xe-0-1-3.edge01.iad02.as13335.net [2001:504:0:2:0:1:3335:1]
  5    20 ms    20 ms    20 ms  2400:cb00:2048:1::681c:c22

Trace complete.


Take a look at these tweets.

https://twitter.com/valeriangalliat/status/559834698130931713

[Napsterbater]

Well I can get to cloudflare sites now, routing kinda sucks but meh, at least its working.

Code Select Expand

Tracing route to sourcefed.com [2400:cb00:2048:1::681c:c22]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  car1.***.local [2001:470:e138:1::3]
  2    24 ms    32 ms    24 ms  ***-2.tunnel.tserv13.ash1.ipv6.he.net [2001:470:7:***::1]
  3    20 ms    20 ms    22 ms  ge4-12.core1.ash1.he.net [2001:470:0:90::1]
  4    33 ms    25 ms    24 ms  100ge5-1.core1.nyc4.he.net [2001:470:0:299::2]
  5    94 ms    98 ms    98 ms  100ge7-2.core1.lon2.he.net [2001:470:0:2cf::1]
  6    92 ms    92 ms    90 ms  2001:7f8:4::329c:1
  7   210 ms   286 ms   202 ms  lo0-grtnycpt2-ip6.red.telefonica-wholesale.net [2001:1498:1::32:132]
  8   206 ms   199 ms   214 ms  lo0-grtmiabr4-ip6.red.telefonica-wholesale.net [2001:1498:1::32:250]
  9   201 ms   209 ms   198 ms  lo0-grtlurem3-ip6.red.telefonica-wholesale.net [2001:1498:1::32:198]
10   194 ms   205 ms   200 ms  CLOUDFARE-1-0-11-0-grtlurem3.ip6.tiws.net [2001:1498:1:795::2]
11   208 ms   198 ms   207 ms  2400:cb00:2048:1::681c:c22

Trace complete.

[uhoreg]

FWIW, the issue looks like it is related to this: https://blog.cloudflare.com/path-mtu-discovery-in-practice/ which hopefuly means that it's fixed now.

[pghe]

My HE tunnel terminates @ the Fremont hub.

I'm seeing the same issues with -> cloudflare !connectivity

Here's the traceroute

Code Select Expand


mtr --show-ips --report-wide --report-cycles=1 cloudflare.com
Start: Thu Mar  5 12:44:39 2015
HOST: xxxx.xxxx.com                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- xxxx.xxxx.com (2001:470:xxxx:xxx::xxx)                         0.0%     1    0.8   0.8   0.8   0.8   0.0
  2.|-- xxxxxxx.tunnel.tserv3.fmt2.ipv6.he.net (2001:470:xxxx:xxx::x)  0.0%     1   49.4  49.4  49.4  49.4   0.0
  3.|-- ge5-19.core1.fmt2.he.net (2001:470:0:45::1)                    0.0%     1   49.8  49.8  49.8  49.8   0.0
  4.|-- 10ge1-1.core1.sjc2.he.net (2001:470:0:31::2)                   0.0%     1   54.5  54.5  54.5  54.5   0.0
  5.|-- 2001:504:0:1:0:1:3335:1                                        0.0%     1   81.0  81.0  81.0  81.0   0.0
  6.|-- ???                                                           100.0     1    0.0   0.0   0.0   0.0   0.0


where

Code Select Expand


host cloudflare.com
cloudflare.com has address 198.41.212.157
cloudflare.com has address 198.41.213.157
cloudflare.com has IPv6 address 2400:cb00:2048:1::c629:d59d
cloudflare.com has IPv6 address 2400:cb00:2048:1::c629:d49d
cloudflare.com mail is handled by 10 aspmx.l.google.com.
cloudflare.com mail is handled by 20 alt1.aspmx.l.google.com.
cloudflare.com mail is handled by 40 aspmx2.googlemail.com.
cloudflare.com mail is handled by 50 aspmx3.googlemail.com.
cloudflare.com mail is handled by 30 alt2.aspmx.l.google.com.

host www.cloudflare.com
www.cloudflare.com is an alias for www.cloudflare.com.cdn.cloudflare.net.
www.cloudflare.com.cdn.cloudflare.net has address 198.41.215.163
www.cloudflare.com.cdn.cloudflare.net has address 198.41.214.163
www.cloudflare.com.cdn.cloudflare.net has IPv6 address 2400:cb00:2048:1::c629:d6a3
www.cloudflare.com.cdn.cloudflare.net has IPv6 address 2400:cb00:2048:1::c629:d7a3