Hello
Are there any known problems with the toronto pop?
It seems I am getting about 60% packetloss via ipv6 the past couple days, just not icmp, but ssh sessions are extremely laggy, and Ive had to disable ipv6 in my browsers as google (and other ipv6 sites) were basically dialup speeds.
ping6 to my ipv6 gateway:
183 packets transmitted, 63 received, 65% packet loss, time 182806ms
rtt min/avg/max/mdev = 29.936/36.440/40.558/2.071 ms
ping4s to the ipv4 endpoint:
-- 216.66.38.58 ping statistics ---
101 packets transmitted, 101 received, 0% packet loss, time 100127ms
rtt min/avg/max/mdev = 8.838/10.851/21.972/1.963 ms
so ipv4 seems fine? but once it goes ipv6, it goes bad. :-(
Nothing has changed on my side that I can think of, I have rebooted my routers, etc, and nothing has changed. Not saying it cant be my equipment, but just weird out of the blue a perfectly working tunnel just starting going bad. :-(
Any ideas?
Thanks
(and I apologize if this shows up twice, I previewed my first post, then went to Post, but it said I already made a post within 5 seconds or something... meh)
mtr from my computer to the IPv4 address of the tunnel server shows zero packet loss. mtr to your IPv6 address shows zero packet loss on the path to the tunnel server and 35% packet loss between the tunnel server and you. So you should take a look on what mtr 216.66.38.58 looks like from your network.
mtr 216.66.38.58 -c 100 --report
HOST: Loss% Snt Last Avg Best Wrst StDev
3.|-- 69.63.242.117 0.0% 100 15.7 14.2 10.2 35.2 4.8
4.|-- 69.63.249.93 0.0% 100 13.3 13.0 9.6 36.2 3.3
5.|-- 10ge7-4.core1.tor1.he.net 0.0% 100 10.1 13.9 8.9 38.6 5.3
6.|-- tserv1.tor1.he.net 0.0% 100 13.9 10.6 9.2 17.3 1.2
mtr google.com -c 50 --report
HOST: Loss% Snt Last Avg Best Wrst StDev
2.|-- grazzt-1.tunnel.tserv21.t 50.0% 50 39.1 39.2 17.4 54.6 7.2
3.|-- ge2-5.core1.tor1.he.net 44.0% 50 43.9 39.1 32.3 48.6 3.9
4.|-- 2001:478:245:1::6 54.0% 50 37.3 38.3 26.9 72.1 8.5
5.|-- 2001:4860::1:0:e38 48.0% 50 34.1 37.3 22.5 67.9 7.3
6.|-- 2001:4860:0:1::71d 42.0% 50 40.7 45.7 21.6 277.8 45.1
7.|-- yyz08s14-in-x03.1e100.net 52.0% 50 34.9 40.5 25.3 134.7 20.8
As you can see, its pretty much perfect on ipv4, but ipv6.. not so much, lol. Thats why Im confused, Im sort of leaning to maybe something on my side, but nothing has changed, and why would it be doing 50-60% packetloss. Maybe I should create a new tunnel to my ip from say, New York and see what happens then.
Thanks for the response.
Quote from: grazzt on January 09, 2014, 06:50:19 AMThats why Im confused, Im sort of leaning to maybe something on my side, but nothing has changed, and why would it be doing 50-60% packetloss.
One possible explanation is that the IPv4 path treats packets differently depending on protocol. Additionally it might be that there are two parallel paths one of which can handle protocol 41 just fine and one, which drops them. Two parallel paths would explain the approximately 50% figure.
Additionally, we don't yet know in which direction the packet loss is happening.
Two ideas, that could give us additional hints about the reason. You could perform a traceroute with protocol 41 packets. With the traceroute version I have installed, the command line to do that would be:
traceroute -n -P41 216.66.38.58Additionally you could take a look on something I wrote recently on detecting in which direction drops are happening. I'd like to know, if that is useful: https://www.tunnelbroker.net/forums/index.php?topic=3069.msg18071#msg18071
Quote from: grazzt on January 09, 2014, 06:50:19 AMMaybe I should create a new tunnel to my ip from say, New York and see what happens then.
Worth trying. But it might not help pinpoint the source of the problem. But don't you already have five tunnels?
traceroute -n -P41 216.66.38.58
traceroute to 216.66.38.58 (216.66.38.58), 64 hops max, 48 byte packets
1 192.168.2.1 0.344 ms 0.312 ms 0.265 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
done on both a freebsd machine and a linux box, always the same results.
Now I am running a tomato based router as 192.168.2.1, maybe its 'blocking' proto 41? (the tomato router has always been there, even when ipv6 was working great for the time Ive been signed up)
I have tried forwarding proto 41 as
iptables -t nat -A WANPREROUTING -p 41 -j DNAT --to-destination lanip
iptables -A wanin -p 41 -d lanip -j ACCEPT
and it doesnt seem to change anything to the traceroutes.
I currently have 4 tunnels, so I can try another. (seems I would have to delete my current toronto one to get a new NY one, I wont do that yet)
And your link is maybe above my head a little, did you want me to compile those programs and run them?
Thanks for the help. :-)
Quote from: grazzt on January 09, 2014, 07:53:00 AMNow I am running a tomato based router as 192.168.2.1, maybe its 'blocking' proto 41?
Where is the tunnel endpoint? If the tunnel endpoint is behind the router, then it couldn't be the router blocking the traffic, because you were after all getting some protocol 41 packets through. However do remember that since you are most likely using NAT for IPv4, you shouldn't expect to be able to have more than one computer talking protocol 41.
The traceroute command I suggested would probably give a much more useful output, if it was run on the router itself. (Assuming the router has a traceroute command with support for the -P option.)
Quote from: grazzt on January 09, 2014, 07:53:00 AMAnd your link is maybe above my head a little, did you want me to compile those programs and run them?
For a start you can try to compile the client program and run it with 2a01:4f8:d16:701:9c65:a13e:5a91:4dd as the server IP address. The output from that should be useful.
You can also compile the server program and tell me the IPv6 address of the host it is running on. That doesn't have to be the router, it could be any computer on the LAN. Then I can run the client from my end to see what results it produces.
The tomato router is the endpoint, it has built in support for ipv6 tunnels, etc.
Unforunately the router, while it does run linux, the traceroute command does not support passing -P option. I suppose it might be possible to install a crosscompiler and compile a specific traceroute. for same reason, cant run the client on the tomato router.
On two linux machines:
/client 2a01:4f8:d16:701:9c65:a13e:5a91:4dd
^C0 0 0 0 Return drop percentage estimates ?
(^C is just an incrementing number)
On a freebsd machine:
./client 2a01:4f8:d16:701:9c65:a13e:5a91:4dd
2a01:4f8:d16:701:9c65:a13e:5a91:4dd: No error: 0
Note all machines exhibit the same problem (50% packet loss)
Im also going to ask my isp about this problem. Im using a third party cable provider, and the incumbent cable provider *loves* to play around with the third party companies.
server is running on both 2001:470:b245:0:6e62:6dff:fe8b:ffa5 (freebsd) and 2001:470:b245:0:224:1dff:fe84:b306 (linux)
There were similar complaints in #ipv6 last night about this. Hopefully someone did more than complain on IRC and forums and actually opened a trouble ticket, by emailing ipv6@he.net, highlighting the IPv6 loss.
Quote from: grazzt on January 09, 2014, 09:14:31 AMOn two linux machines:
/client 2a01:4f8:d16:701:9c65:a13e:5a91:4dd
^C0 0 0 0 Return drop percentage estimates ?
That means it didn't manage to get a single packet through. Looks like a three digit number, so that means you did let it run until the counter increased beyond 100?
Quote from: grazzt on January 09, 2014, 09:14:31 AM(^C is just an incrementing number)
Some stupidity in bash or readline. Older Linux systems did not mangle the output.
Quote from: grazzt on January 09, 2014, 09:14:31 AMOn a freebsd machine:
./client 2a01:4f8:d16:701:9c65:a13e:5a91:4dd
2a01:4f8:d16:701:9c65:a13e:5a91:4dd: No error: 0
Looks like my code does not work on BSD then. Different behaviour of getaddrinfo as far as I can tell.
Quote from: grazzt on January 09, 2014, 09:14:31 AMserver is running on both 2001:470:b245:0:6e62:6dff:fe8b:ffa5 (freebsd) and 2001:470:b245:0:224:1dff:fe84:b306 (linux)
I didn't get any packets through to either. Could it be that your router has a packet filter, which drops ICMPv6 packets of unknown type?
The server on 2a01:4f8:d16:701:9c65:a13e:5a91:4dd did not receive any packets from the client program. It did however receive ordinary echo requests from your Linux machine. Is that because you were running a ping command?
Quote from: broquea on January 09, 2014, 09:57:11 AMThere were similar complaints in #ipv6 last night about this. Hopefully someone did more than complain on IRC and forums and actually opened a trouble ticket, by emailing ipv6@he.net, highlighting the IPv6 loss.
Until your posting this thread looked like the problem might just as well be on the IPv4 path between client and server. If another user had confirmed that he saw the problem as well, I would have recommended a ticket be opened. When a service like tunnelbroker.net is provided for free, I feel obliged to do a bit more debugging before filing a ticket. I don't open a ticket before I have convinced myself, that the problem is on the HE side. But maybe that is just me.
(If I weren't limited to at most five tunnels, and at most one tunnel per client IPv4 address, I would happily configure a tunnel per tunnel server, such that I could try to reproduce problems experienced by other users, and carefully explain my findings in a ticket.)
It is really on the the user experiencing the issue to do the reporting, you shouldn't feel obligated to duplicate and complain on their behalf. This user could be the same that was on IRC, no idea. Tickets will garner more attention that hoping someone reads the forum that day. They are paid to read the tickets :)
Unknown about the router filtering unknown ipv6 packets
Packet loss is now down to 6%, so maybe its going to fix itself?? (and now 1.7%, so maybe improving)
And once again, thanks for all the help. :-)
And no, I wasnt the one on irc complaining. I thought I would post here first, maybe should have just opened a ticket. <shrug>. And the problem just didnt start today, its been like this for 3 days? I sort of waited for it to fix itself, seemed like it didnt, came here, and now it seems to be better..
sigh, and now as i try again, its at 30%. who knows. lol
Quote from: broquea on January 09, 2014, 10:49:17 AMThey are paid to read the tickets :)
Not by the users, but perhaps by somebody else. Since I am not paying for the service, I don't feel entitled to getting support from them without even looking into the problem and providing the relevant input I can provide.
Well shit, I should have ignored all the tickets when I ran the service then :D There were tons of tickets that never came close to providing the level of info you look up, and were wayyyyyy more demanding of better service "or their money back".
Quote from: broquea on January 09, 2014, 03:43:44 PM
Well shit, I should have ignored all the tickets when I ran the service then :D There were tons of tickets that never came close to providing the level of info you look up, and were wayyyyyy more demanding of better service "or their money back".
I won't say I am surprised. But you'll never see me suggesting users file a ticket without first collecting available information that would help pinpoint the location of the problem. Anyway, I guess the users you are referring to rarely consulted the forum before filing a ticket.
The way I see it, the forum should be useful:
- If you want help learning how to track the problem down.
- If you want advice on how to interpret the data you collected.
- If you want to know if other people are affected by the same problem.
But once you have reached the conclusion that the problem is on the HE network or getting any closer to an explanation needs information that only an administrator on the HE network could see, then you should file a ticket and not expect a forum post to reach those people at HE, who can help.
fwiw, switched to a NY tunnel, and its the same packet loss, so its either something messed up on my end, or my isp is doing something messed up. (although for about an hour there, there was minimal loss, so shrug). Actually running at about 80-85% loss now. :-/
Next step will be to eliminate the tomato router and directly connect to a linux or bsd box and see what happens.
Hi.
I'm having same issue with Toronto server for last 3-4 days. Connected with Teksavvy via Cogeco. If I didn't misread anything original poster was using Roges, i.e. different ISP.
I'm using OpenWRT router and have been using it for years without issues.
The packet loss occurs even if I ping 'server ipv6 address', i.e. closest ipv6 address I can get.
Didn't try other tunnel servers though. Filed a ticket with HE.
its fixed now, i think it was an ISP problem.
Thanks for all those who suggested solutions.
Doesn't seem like fixed to me :(.
I'm still getting about 5% of pings lost when I ping ipv6 address on HE's end of the tunnel.
Quote from: markolya on January 11, 2014, 05:22:43 PMI'm still getting about 5% of pings lost when I ping ipv6 address on HE's end of the tunnel.
Is your tunnel up? I get no replies at all, when I try to ping your tunnel endpoint.
Quote from: kasperd on January 12, 2014, 06:41:40 AM
Quote from: markolya on January 11, 2014, 05:22:43 PMI'm still getting about 5% of pings lost when I ping ipv6 address on HE's end of the tunnel.
Is your tunnel up? I get no replies at all, when I try to ping your tunnel endpoint.
Yes, judging by the time of your message that tunnel should have been up. It looks like it is better this morning - no losses so far. I guess I'll monitor it for while and reply if there will be problems.
Thanks!
Quote from: markolya on January 12, 2014, 10:00:38 AMYes, judging by the time of your message that tunnel should have been up. It looks like it is better this morning - no losses so far. I guess I'll monitor it for while and reply if there will be problems.
Looks responsive now. Maybe HE were restarting the tunnel server at the time I was testing. If anybody filed a ticket, they should know, if HE did in fact restart the tunnel server to resolve the problem.
Well, it looks like it is better now - no packet loss, but there's still something weird going on:
When I ping ipv4 of the tunnel's endpoint from y router I get something like this:
64 bytes from 216.66.38.58: seq=0 ttl=59 time=13.443 ms
64 bytes from 216.66.38.58: seq=1 ttl=59 time=17.160 ms
64 bytes from 216.66.38.58: seq=2 ttl=59 time=14.440 ms
64 bytes from 216.66.38.58: seq=3 ttl=59 time=15.821 ms
64 bytes from 216.66.38.58: seq=4 ttl=59 time=15.491 ms
I.e. it is about 16ms +- 2ms.
I get similar results when I ping google over ipv4. Today, Dusing the day I was getting similar results when I was pinging google over ipv6.
Now, in the evening I get this:
64 bytes from 2001:470:1c:77a::1: seq=12 ttl=64 time=78.401 ms
64 bytes from 2001:470:1c:77a::1: seq=13 ttl=64 time=44.290 ms
64 bytes from 2001:470:1c:77a::1: seq=14 ttl=64 time=49.067 ms
64 bytes from 2001:470:1c:77a::1: seq=15 ttl=64 time=60.406 ms
64 bytes from 2001:470:1c:77a::1: seq=16 ttl=64 time=53.259 ms
(here I'm pinging tunnel's HE's side endpoint from my router).
Some times it goes up to 100ms so it is both longer and generally less stable.
Note: ping to 216.66.38.58 is still about 16ms, at the same time.
So this is kind of weird. My (very nooby) understanding is that 216.66.38.58 and 2001:470:1c:77a::1 should be physically same piece of hardware, so ping time should not differ much.
But in fact it does, and it looks like it starts to happen more during 'peak hours' (Sunday evening). Not sure what exactly could be going on here and how to diagnose this :(.
Yeah, I've created a ticket with HE, but have not heard from them yet... and I'm not sure there will actually response to me :).
Quote from: markolya on January 12, 2014, 03:27:27 PMMy (very nooby) understanding is that 216.66.38.58 and 2001:470:1c:77a::1 should be physically same piece of hardware, so ping time should not differ much.
Yeah, that's what one would think. But how do you count pieces of hardware? The physical distance between them is not necessarily related to how tightly they are coupled in operation. I have seen hardware split across different cabinets acting as one computer. And I have seen cases where two boards in the same cabinet are practically independent computers.
Backbone routers tend to be designed to handle lots of packets in hardware. Decrementing hop limit and forwarding onto another interface is definitely one of those things, which can be done in hardware. Encapsulating and decapsulating protocol 41 might be doable in hardware, but there is probably less hardware, which supports that operation.
Those packets which cannot be handled by the hardware are then to be handled by a CPU. The routing hardware can handle a lot more packets per second than the CPU, but what fraction of the packets send to the router can be handled by the hardware depends on what actual traffic looks like. This means we don't know exactly which packets need to the CPU and which do not. We also don't know how busy either is. Most people would probably consider a CPU and a routing/switching chip mounted on a single board to be one piece of hardware, but ping time between the two could differ.
I have no idea what hardware platform tunnelbroker.net is based on. However I have noticed some differences in behaviour between the different tunnel servers, which leads me to guess, that they are not all identical.
So many unknowns. And it takes so little to affect the timings you measure. So in the end it is hard to figure out, why you see the delays you do and if that even matters.
I suggest you try using mtr (or some similar tool) to measure roundtrip not only to the first hop, but to multiple hops on a path from you to some well-connected IPv6 site. If only the tunnel server itself is showing variations in roundtrip time, but some later hop shows consistent latency, then it might be due to the replies from the tunnel server being bottlenecked by CPU, but replies from later hops not going through the CPU on the tunnel server.
Quote from: kasperd on January 12, 2014, 04:44:45 PM
I suggest you try using mtr (or some similar tool) to measure roundtrip not only to the first hop, but to multiple hops on a path from you to some well-connected IPv6 site. If only the tunnel server itself is showing variations in roundtrip time, but some later hop shows consistent latency, then it might be due to the replies from the tunnel server being bottlenecked by CPU, but replies from later hops not going through the CPU on the tunnel server.
I did that. I tried google - I see same response time fluctuations, starting at first hop.
I also tried pinging those ipv4 and ipv6 from 'some server' in another part of the world. Ping time was in 80ms range due to distance. Fluctuations were in a range of 5ms. But most importantly difference between ipv4 and ipv6 was about 2ms, probably because 'some server' uses different paths for ipv4 and ipv6 traffic. But it definitely was not 50ms I see when I ping from tunnel...