Latency to google.com via he.net vs. CenturyLink 6rd

[wildlava]

Hurricane Electric's tunnel broker service is great, and it's so cool to have IPv6 working on my network!  There is just one thing I would like to see improved: latency to Google.  Since it is used by many for basic and everyday services, and since Google has network presence so many places (and I read that he.net's connection to Google is pretty direct), I'd expect a pretty short path/ping there.  I compared he.net's tunnel to CenturyLink's 6rd tunnel, and I was surprized to see these results:

CenturyLink
---------------
PING google.com(den03s06-in-x0e.1e100.net) 56 data bytes
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=1 ttl=57 time=21.8 ms
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=2 ttl=57 time=22.0 ms
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=3 ttl=57 time=21.7 ms
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=4 ttl=57 time=22.2 ms
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=5 ttl=57 time=21.9 ms

traceroute to google.com (2607:f8b0:400f:801::100e), 30 hops max, 80 byte packets
1  myrouter.mydomain.com (2602:xx:xxxx:xx00::1)  0.299 ms  0.280 ms  0.339 ms
2  2602::205:171:3:251 (2602::205:171:3:251)  20.878 ms  20.890 ms  20.874 ms
3  2001:428:0:1:205:171:253:81 (2001:428:0:1:205:171:253:81)  20.856 ms  20.840 ms  20.900 ms
4  2001:428::205:171:202:212 (2001:428::205:171:202:212)  22.411 ms  22.278 ms  22.142 ms
5  2001:428:3801:210:0:2:0:2 (2001:428:3801:210:0:2:0:2)  22.144 ms  22.132 ms  22.211 ms
6  2001:4860::1:0:7a4 (2001:4860::1:0:7a4)  22.406 ms  21.836 ms  21.793 ms
7  2001:4860:0:1::593 (2001:4860:0:1::593)  22.123 ms  21.243 ms  21.711 ms
8  2607:f8b0:8000:1d::8 (2607:f8b0:8000:1d::8)  21.163 ms  21.432 ms  21.444 ms

Hurricane Electric
----------------------
PING google.com(den03s06-in-x02.1e100.net) 56 data bytes
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=1 ttl=52 time=68.6 ms
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=2 ttl=52 time=68.5 ms
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=3 ttl=52 time=68.6 ms
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=4 ttl=52 time=68.5 ms
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=5 ttl=52 time=68.5 ms

traceroute to google.com (2607:f8b0:400f:801::1002), 30 hops max, 80 byte packets
1  myrouter.mydomain.com (2001:470:xxxx::1)  0.265 ms  0.279 ms  0.384 ms
2  endpoint-1.tunnel.tserv1.den1.ipv6.he.net (2001:470:xxxx:xxxx::1)  21.397 ms  21.414 ms  21.669 ms
3  gige-g2-19.core1.den1.he.net (2001:470:0:24b::1)  21.981 ms  21.987 ms  22.131 ms
4  10gigabitethernet4-3.core1.chi1.he.net (2001:470:0:1af::1)  46.888 ms  47.419 ms  47.567 ms
5  google-as15169.10gigabitethernet7.switch2.chi1.he.net (2001:470:0:bf::2)  46.031 ms  46.015 ms  46.119 ms
6  2001:4860::1:0:3f7 (2001:4860::1:0:3f7)  45.906 ms 2001:4860::1:0:92e (2001:4860::1:0:92e)  45.065 ms 2001:4860::1:0:3f7 (2001:4860::1:0:3f7)  44.587 ms
7  2001:4860::8:0:2fe9 (2001:4860::8:0:2fe9)  45.561 ms  45.942 ms  45.883 ms
8  2001:4860::8:0:281d (2001:4860::8:0:281d)  58.498 ms  56.057 ms 2001:4860::8:0:281e (2001:4860::8:0:281e)  56.007 ms
9  2001:4860::8:0:3426 (2001:4860::8:0:3426)  66.460 ms 2001:4860::8:0:3427 (2001:4860::8:0:3427)  67.870 ms 2001:4860::8:0:3426 (2001:4860::8:0:3426)  67.857 ms
10  2001:4860::1:0:7a4 (2001:4860::1:0:7a4)  66.433 ms  66.417 ms  68.742 ms
11  2001:4860:0:1::593 (2001:4860:0:1::593)  69.215 ms  67.099 ms  67.160 ms
12  2607:f8b0:8000:1d::8 (2607:f8b0:8000:1d::8)  66.872 ms  66.860 ms  66.879 ms

I am in Denver, and there appears to be an IPv6 Google node in Denver as well.  Both tunnels have endpoints in Denver, so from each tunnel endpoint to Google can be a very short path.  With CenturyLink, latency is not much more than to the endpoint, whereas with he.net, it seems to bounce around a bit more, perhaps even visiting Chicago (? not sure if that is where chi1.he.net is) before ending up at Google, and latency is more than 3 times as long.

Anyone know what is going on?  Can this path be made more direct in he.net's routing tables?

Thanks, Joe

[aaron465]

I am seeing a very high latency to google also, I am in the UK and connected to London tunnel server but I believe that Google is geo-locating by the IPv6 address and sending us to a US server - weird thing is we have high latency to google on IPv4 as well but all other sites seem fine. Ping below for v6 & v4 to a number of sites (trouter.co.uk is my own website on Rackspace cloud UK).

Weird thing though - the pings below are from my Windows PC, but when I ping from a Debian server on the same network I get sub 40ms to google.co.uk / .com on both IPv4 and v6. Other Windows machines on the network are seeing high latency to google also.

C:\Users\Aaron>ping -4 google.co.uk

Pinging google.co.uk [74.125.237.159] with 32 bytes of data:
Reply from 74.125.237.159: bytes=32 time=366ms TTL=45
Reply from 74.125.237.159: bytes=32 time=361ms TTL=45
Reply from 74.125.237.159: bytes=32 time=364ms TTL=45
Reply from 74.125.237.159: bytes=32 time=359ms TTL=45

Ping statistics for 74.125.237.159:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 359ms, Maximum = 366ms, Average = 362ms

C:\Users\Aaron>ping -6 google.co.uk

Pinging google.co.uk [2404:6800:4006:804::101f] with 32 bytes of data:
Reply from 2404:6800:4006:804::101f: time=371ms
Reply from 2404:6800:4006:804::101f: time=367ms
Reply from 2404:6800:4006:804::101f: time=366ms
Reply from 2404:6800:4006:804::101f: time=370ms

Ping statistics for 2404:6800:4006:804::101f:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 366ms, Maximum = 371ms, Average = 368ms

C:\Users\Aaron>ping -4 trouter.co.uk

Pinging trouter.co.uk [95.138.172.25] with 32 bytes of data:
Reply from 95.138.172.25: bytes=32 time=20ms TTL=52
Reply from 95.138.172.25: bytes=32 time=36ms TTL=52
Reply from 95.138.172.25: bytes=32 time=20ms TTL=52
Reply from 95.138.172.25: bytes=32 time=18ms TTL=52

Ping statistics for 95.138.172.25:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 18ms, Maximum = 36ms, Average = 23ms

C:\Users\Aaron>ping -6 trouter.co.uk

Pinging trouter.co.uk [2a00:1a48:7805:113:579c:7b0c:ff08:39e4] with 32 bytes of data:
Reply from 2a00:1a48:7805:113:579c:7b0c:ff08:39e4: time=40ms
Reply from 2a00:1a48:7805:113:579c:7b0c:ff08:39e4: time=35ms
Reply from 2a00:1a48:7805:113:579c:7b0c:ff08:39e4: time=41ms
Reply from 2a00:1a48:7805:113:579c:7b0c:ff08:39e4: time=34ms

Ping statistics for 2a00:1a48:7805:113:579c:7b0c:ff08:39e4:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 34ms, Maximum = 41ms, Average = 37ms

C:\Users\Aaron>ping -4 bbc.co.uk

Pinging bbc.co.uk [212.58.251.195] with 32 bytes of data:
Reply from 212.58.251.195: bytes=32 time=18ms TTL=118
Reply from 212.58.251.195: bytes=32 time=20ms TTL=118
Reply from 212.58.251.195: bytes=32 time=18ms TTL=118
Reply from 212.58.251.195: bytes=32 time=21ms TTL=118

Ping statistics for 212.58.251.195:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 18ms, Maximum = 21ms, Average = 19ms

[kasperd]

I am seeing a very high latency to google also, I am in the UK and connected to London tunnel server but I believe that Google is geo-locating by the IPv6 address and sending us to a US server

Reverse DNS on the IP sounds like Sydney.

Also, notice that this has nothing to do with geolocation of any IPv6 address. Until Google gets full IPv6 support, they cannot direct you to a server based on IPv6 geolocation.

Your problem is, that you are using a DNS server, which Google thinks is in Sydney. Try using another DNS server.

[aaron465]

Using google public DNS :P

stateless DHCP configured to give out their v6 addresses also. I did think 300ms was a bit high even for USA. Any pointers on fixing this?

[kasperd]

Using google public DNS

It is pretty messed up, if Google can't keep track of where in the world their own DNS servers are.

When both the recursors and the authoritative DNS servers are operated by Google, they can take advantage of that in order to have more data to use in the geolocation. In that case they could actually base it on both the client IPv6 address and the location of the DNS recursor.

Since Google Public DNS is using anycast IP addresses, it is almost guaranteed, that clients will be using a DNS recursor close to themselves. Thus which DNS server you are using would be a more reliable indication of your location than your IP address (because the later relies on the accuracy of the geolocation database).

I managed to reproduce the problem myself. This is sufficiently messed up, that I'll try to email one of my contacts at Google directly. Hopefully that way we can find out, what is going on.

[aaron465]

Hmm ok might switch to opendns for the time being then - probably the same thing causing issues with the Google IO live stream the other day, I ended up watching over 3G on my tablet  :P

[kasperd]

might switch to opendns for the time being then

Given opendns' history of hijacking some requests by handing out their own IP address instead of the correct IP address for the domain, I wouldn't consider them. How about using the HE anycast DNS address as your primary DNS server, I think geolocation is done correctly for the HE DNS servers.

I did a bit more digging into the problem. The problem is not only with google.co.uk. I could reproduce the problem with google.com as well. Moreover I found that there was only a problem if I send requests to Google Public DNS over IPv6. If I did it over IPv4, I got a proper address back.

I tried from my laptop through two different tunnel brokers and 6to4. I also tried over native IPv6 from my VPS. I could only reproduce the problem through the HE tunnel. That means either the HE tunnel is reaching a different instance of Google Public DNS, or they are doing geolocation based on client IP. My guess was that it was based on client IP.

To verify this I needed to query DNS using different client addresses, but in a way that uses the same network path from my computer to the DNS server. Due to ingress filtering, that is not trivial to do. But I happen to know a trick that allows me to send IPv6 packets through my ISP with arbitrary source address. (This works in spite of this ISP not even offering IPv6 yet). That way I was able to send queries to the same DNS server with different source addresses.

I found that the DNS server I reached would only give me an IP address in Sydney, if my client address was from the HE range. So Google appears to be doing the geolocation based on client IP rather than proximity to the DNS server.

I don't know if the geolocation as Sydney applies to all HE addresses or only some of them. For now we know that users of the London and Frankfurt tunnel servers are affected. This problem is BTW completely unrelated to the problem, which was being asked at the start of this thread. That question remains unanswered.

[aaron465]

I have left the v4 name servers set as google and changed the v6 to the HE one. Seems to be working a lot better now, and Youtube seems to buffer properly where it was having issues before :)

Thanks for the insight into this.

Aaron

[wildlava]

In my case it is not DNS-related.  The final Google IP is the same for both traceroutes (through CenturyLink and through HE).  Weird that there are 4 more hops with HE (and more latency).

Since the Google presence in Denver is a very short hop from the CenturyLink tunnel endpoint in Denver, and the HE endpoint is also in Denver, I'd think the HE tunnel endpoint would also be a very short hop to Denver's Google node.  Looks like it's routing, instead, to unnecessarily far nodes.

[kasperd]

In my case it is not DNS-related.

True. I noticed that much, when I read your post the first time. However since I didn't have anything to add to what you already found out yourself, I didn't reply at the time. You could write to ipv6@he.net and ask if there is some reason for the length of that path. I guess a roundtrip time of 67ms just isn't considered to be long enough to really be a concern.

If Google and HE are peering in both locations it is difficult for an outsider to find out who chooses which of the peering points will be used. It is also possible that the traffic takes different paths in the two directions. A traceroute will only reveal the forward path, and the timings on the traceroute will include both forward and return path with no way to know, how much time was used in each direction. It is even possible, that the different hops you see on the traceroute will take different return paths.

[wildlava]

Does anyone know (or know how to find out) where chi1.he.net is?  This snippet from the traceroute to Google shows packets being routed from Denver to there and then to what looks like the peering to Google:

3  gige-g2-19.core1.den1.he.net (2001:470:0:24b::1)  21.981 ms  21.987 ms  22.131 ms
4  10gigabitethernet4-3.core1.chi1.he.net (2001:470:0:1af::1)  46.888 ms  47.419 ms  47.567 ms
5  google-as15169.10gigabitethernet7.switch2.chi1.he.net (2001:470:0:bf::2)  46.031 ms  46.015 ms  46.119 ms

Can others reading this try their traceroute to Google and see if it also goes through chi1.he.net?  I am curious if all locations in the US (or world) go that route.

It was only a guess that "chi" is Chicago...  But since Google has presence in Denver (as evidenced by the CenturyLink path), could this long haul be avoided?  I'm not sure how difficult it is to always route to the nearest Google node rather than some other one far away, but Google's network architecture (that makes Google response time so snappy and makes network load distributed) seems to be designed to have everyone get to their nearest Google node rather than traversing the country to get to them.

A ping in the 60s is not horrible, but many of Google's services are highly interactive (Ajax-based web apps, etc.), so a ping time of 20ms is a noticeable improvement.  I prefer he.net's tunnel to CenturyLink's, but I currently incur the latency penalty with he.net, unfortunately, which is a bummer.

[broquea]

chi = chicago

[wildlava]

chi = chicago

Thanks!  OK, so I wonder if this means Chicago is the only Google peering for he.net...?

[kasperd]

I wonder if this means Chicago is the only Google peering for he.net

Definitely not, when I do a traceroute, I see a peering in Germany. The question to ask is whether there is a POP closer to Denver, where both HE and Google have a presence, or if Chicago is the best option.

[wildlava]

Hurricane Electric's tunnel broker service is great, and it's so cool to have IPv6 working on my network!  There is just one thing I would like to see improved: latency to Google.  Since it is used by many for basic and everyday services, and since Google has network presence so many places (and I read that he.net's connection to Google is pretty direct), I'd expect a pretty short path/ping there.  I compared he.net's tunnel to CenturyLink's 6rd tunnel, and I was surprized to see these results:

CenturyLink
---------------
PING google.com(den03s06-in-x0e.1e100.net) 56 data bytes
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=1 ttl=57 time=21.8 ms
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=2 ttl=57 time=22.0 ms
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=3 ttl=57 time=21.7 ms
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=4 ttl=57 time=22.2 ms
64 bytes from den03s06-in-x0e.1e100.net: icmp_seq=5 ttl=57 time=21.9 ms

traceroute to google.com (2607:f8b0:400f:801::100e), 30 hops max, 80 byte packets
1  myrouter.mydomain.com (2602:xx:xxxx:xx00::1)  0.299 ms  0.280 ms  0.339 ms
2  2602::205:171:3:251 (2602::205:171:3:251)  20.878 ms  20.890 ms  20.874 ms
3  2001:428:0:1:205:171:253:81 (2001:428:0:1:205:171:253:81)  20.856 ms  20.840 ms  20.900 ms
4  2001:428::205:171:202:212 (2001:428::205:171:202:212)  22.411 ms  22.278 ms  22.142 ms
5  2001:428:3801:210:0:2:0:2 (2001:428:3801:210:0:2:0:2)  22.144 ms  22.132 ms  22.211 ms
6  2001:4860::1:0:7a4 (2001:4860::1:0:7a4)  22.406 ms  21.836 ms  21.793 ms
7  2001:4860:0:1::593 (2001:4860:0:1::593)  22.123 ms  21.243 ms  21.711 ms
8  2607:f8b0:8000:1d::8 (2607:f8b0:8000:1d::8)  21.163 ms  21.432 ms  21.444 ms

Hurricane Electric
----------------------
PING google.com(den03s06-in-x02.1e100.net) 56 data bytes
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=1 ttl=52 time=68.6 ms
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=2 ttl=52 time=68.5 ms
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=3 ttl=52 time=68.6 ms
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=4 ttl=52 time=68.5 ms
64 bytes from den03s06-in-x02.1e100.net: icmp_seq=5 ttl=52 time=68.5 ms

traceroute to google.com (2607:f8b0:400f:801::1002), 30 hops max, 80 byte packets
1  myrouter.mydomain.com (2001:470:xxxx::1)  0.265 ms  0.279 ms  0.384 ms
2  endpoint-1.tunnel.tserv1.den1.ipv6.he.net (2001:470:xxxx:xxxx::1)  21.397 ms  21.414 ms  21.669 ms
3  gige-g2-19.core1.den1.he.net (2001:470:0:24b::1)  21.981 ms  21.987 ms  22.131 ms
4  10gigabitethernet4-3.core1.chi1.he.net (2001:470:0:1af::1)  46.888 ms  47.419 ms  47.567 ms
5  google-as15169.10gigabitethernet7.switch2.chi1.he.net (2001:470:0:bf::2)  46.031 ms  46.015 ms  46.119 ms
6  2001:4860::1:0:3f7 (2001:4860::1:0:3f7)  45.906 ms 2001:4860::1:0:92e (2001:4860::1:0:92e)  45.065 ms 2001:4860::1:0:3f7 (2001:4860::1:0:3f7)  44.587 ms
7  2001:4860::8:0:2fe9 (2001:4860::8:0:2fe9)  45.561 ms  45.942 ms  45.883 ms
8  2001:4860::8:0:281d (2001:4860::8:0:281d)  58.498 ms  56.057 ms 2001:4860::8:0:281e (2001:4860::8:0:281e)  56.007 ms
9  2001:4860::8:0:3426 (2001:4860::8:0:3426)  66.460 ms 2001:4860::8:0:3427 (2001:4860::8:0:3427)  67.870 ms 2001:4860::8:0:3426 (2001:4860::8:0:3426)  67.857 ms
10  2001:4860::1:0:7a4 (2001:4860::1:0:7a4)  66.433 ms  66.417 ms  68.742 ms
11  2001:4860:0:1::593 (2001:4860:0:1::593)  69.215 ms  67.099 ms  67.160 ms
12  2607:f8b0:8000:1d::8 (2607:f8b0:8000:1d::8)  66.872 ms  66.860 ms  66.879 ms

I am in Denver, and there appears to be an IPv6 Google node in Denver as well.  Both tunnels have endpoints in Denver, so from each tunnel endpoint to Google can be a very short path.  With CenturyLink, latency is not much more than to the endpoint, whereas with he.net, it seems to bounce around a bit more, perhaps even visiting Chicago (? not sure if that is where chi1.he.net is) before ending up at Google, and latency is more than 3 times as long.

Anyone know what is going on?  Can this path be made more direct in he.net's routing tables?

Any updates on this? Is there any way to get directly to Denver Google without going Denver->Chicago->Google?

Thanks, Joe