Problems in toronto?

[grazzt]

fwiw, switched to a NY tunnel, and its the same packet loss, so its either something messed up on my end, or my isp is doing something messed up.  (although for about an hour there, there was minimal loss, so shrug).  Actually running at about 80-85% loss now. :-/

Next step will be to eliminate the tomato router and directly connect to a linux or bsd box and see what happens.

[markolya]

Hi.

 I'm having same issue with Toronto server for last 3-4 days. Connected with Teksavvy via Cogeco. If I didn't misread anything original poster was using Roges, i.e. different ISP.

 I'm using OpenWRT router and have been using it for years without issues.

 The packet loss occurs even if I ping 'server ipv6 address', i.e. closest ipv6 address I can get.

 Didn't try other tunnel servers though. Filed a ticket with HE.

[grazzt]

its fixed now, i think it was an ISP problem.

Thanks for all those who suggested solutions.

[markolya]

Doesn't seem like fixed to me :(.

I'm still getting about 5% of pings lost when I ping ipv6 address on HE's end of the tunnel.

[kasperd]

I'm still getting about 5% of pings lost when I ping ipv6 address on HE's end of the tunnel.

Is your tunnel up? I get no replies at all, when I try to ping your tunnel endpoint.

[markolya]

I'm still getting about 5% of pings lost when I ping ipv6 address on HE's end of the tunnel.

Is your tunnel up? I get no replies at all, when I try to ping your tunnel endpoint.

Yes, judging by the time of your message that tunnel should have been up. It looks like it is better this morning - no losses so far. I guess I'll monitor it for while and reply if there will be problems.

Thanks!

[kasperd]

Yes, judging by the time of your message that tunnel should have been up. It looks like it is better this morning - no losses so far. I guess I'll monitor it for while and reply if there will be problems.

Looks responsive now. Maybe HE were restarting the tunnel server at the time I was testing. If anybody filed a ticket, they should know, if HE did in fact restart the tunnel server to resolve the problem.

[markolya]

Well, it looks like it is better now - no packet loss, but there's still something weird going on:

When I ping ipv4 of the tunnel's endpoint from y router I get something like this:

64 bytes from 216.66.38.58: seq=0 ttl=59 time=13.443 ms
64 bytes from 216.66.38.58: seq=1 ttl=59 time=17.160 ms
64 bytes from 216.66.38.58: seq=2 ttl=59 time=14.440 ms
64 bytes from 216.66.38.58: seq=3 ttl=59 time=15.821 ms
64 bytes from 216.66.38.58: seq=4 ttl=59 time=15.491 ms

I.e. it is about 16ms +- 2ms.

I get similar results when I ping google over ipv4. Today, Dusing the day I was getting similar results when I was pinging google over ipv6.

Now, in the evening I get this:

64 bytes from 2001:470:1c:77a::1: seq=12 ttl=64 time=78.401 ms
64 bytes from 2001:470:1c:77a::1: seq=13 ttl=64 time=44.290 ms
64 bytes from 2001:470:1c:77a::1: seq=14 ttl=64 time=49.067 ms
64 bytes from 2001:470:1c:77a::1: seq=15 ttl=64 time=60.406 ms
64 bytes from 2001:470:1c:77a::1: seq=16 ttl=64 time=53.259 ms

(here I'm pinging tunnel's HE's side endpoint from my router).
Some times it goes up to 100ms so it is both longer and generally less stable.
Note: ping to 216.66.38.58 is still about 16ms, at the same time.

So this is kind of weird. My (very nooby) understanding is that 216.66.38.58 and 2001:470:1c:77a::1 should be physically same piece of hardware, so ping time should not differ much.
But in fact it does, and it looks like it starts to happen more during 'peak hours' (Sunday evening). Not sure what exactly could be going on here and how to diagnose this :(.

Yeah, I've created a ticket with HE, but have not heard from them yet... and I'm not sure there will actually response to me :).

[kasperd]

My (very nooby) understanding is that 216.66.38.58 and 2001:470:1c:77a::1 should be physically same piece of hardware, so ping time should not differ much.

Yeah, that's what one would think. But how do you count pieces of hardware? The physical distance between them is not necessarily related to how tightly they are coupled in operation. I have seen hardware split across different cabinets acting as one computer. And I have seen cases where two boards in the same cabinet are practically independent computers.

Backbone routers tend to be designed to handle lots of packets in hardware. Decrementing hop limit and forwarding onto another interface is definitely one of those things, which can be done in hardware. Encapsulating and decapsulating protocol 41 might be doable in hardware, but there is probably less hardware, which supports that operation.

Those packets which cannot be handled by the hardware are then to be handled by a CPU. The routing hardware can handle a lot more packets per second than the CPU, but what fraction of the packets send to the router can be handled by the hardware depends on what actual traffic looks like. This means we don't know exactly which packets need to the CPU and which do not. We also don't know how busy either is. Most people would probably consider a CPU and a routing/switching chip mounted on a single board to be one piece of hardware, but ping time between the two could differ.

I have no idea what hardware platform tunnelbroker.net is based on. However I have noticed some differences in behaviour between the different tunnel servers, which leads me to guess, that they are not all identical.

So many unknowns. And it takes so little to affect the timings you measure. So in the end it is hard to figure out, why you see the delays you do and if that even matters.

I suggest you try using mtr (or some similar tool) to measure roundtrip not only to the first hop, but to multiple hops on a path from you to some well-connected IPv6 site. If only the tunnel server itself is showing variations in roundtrip time, but some later hop shows consistent latency, then it might be due to the replies from the tunnel server being bottlenecked by CPU, but replies from later hops not going through the CPU on the tunnel server.

[markolya]

I suggest you try using mtr (or some similar tool) to measure roundtrip not only to the first hop, but to multiple hops on a path from you to some well-connected IPv6 site. If only the tunnel server itself is showing variations in roundtrip time, but some later hop shows consistent latency, then it might be due to the replies from the tunnel server being bottlenecked by CPU, but replies from later hops not going through the CPU on the tunnel server.

I did that. I tried google - I see same response time fluctuations, starting at first hop.
I also tried pinging those ipv4 and ipv6 from 'some server' in another part of the world. Ping time was in 80ms range due to distance. Fluctuations were in a range of 5ms. But most importantly difference between ipv4 and ipv6 was about 2ms, probably because 'some server' uses different paths for ipv4 and ipv6 traffic. But it definitely was not 50ms I see when I ping from tunnel...