Problems in toronto?

[grazzt]

Hello

Are there any known problems with the toronto pop?

It seems I am getting about 60% packetloss via ipv6 the past couple days, just not icmp, but ssh sessions are extremely laggy, and Ive had to disable ipv6 in my browsers as google (and other ipv6 sites) were basically dialup speeds.

ping6 to my ipv6 gateway:

183 packets transmitted, 63 received, 65% packet loss, time 182806ms
rtt min/avg/max/mdev = 29.936/36.440/40.558/2.071 ms

ping4s to the ipv4 endpoint:

-- 216.66.38.58 ping statistics ---
101 packets transmitted, 101 received, 0% packet loss, time 100127ms
rtt min/avg/max/mdev = 8.838/10.851/21.972/1.963 ms

so ipv4 seems fine? but once it goes ipv6, it goes bad. :-(

Nothing has changed on my side that I can think of, I have rebooted my routers, etc, and nothing has changed.  Not saying it cant be my equipment, but just weird out of the blue a perfectly working tunnel just starting going bad.  :-(

Any ideas?

Thanks

(and I apologize if this shows up twice, I previewed my first post, then went to Post, but it said I already made a post within 5 seconds or something... meh)

[kasperd]

mtr from my computer to the IPv4 address of the tunnel server shows zero packet loss. mtr to your IPv6 address shows zero packet loss on the path to the tunnel server and 35% packet loss between the tunnel server and you. So you should take a look on what mtr 216.66.38.58 looks like from your network.

[grazzt]

mtr 216.66.38.58 -c 100 --report
HOST:           Loss%   Snt   Last   Avg  Best  Wrst StDev

 3.|-- 69.63.242.117              0.0%   100   15.7  14.2  10.2  35.2   4.8
 4.|-- 69.63.249.93      0.0%   100   13.3  13.0   9.6  36.2   3.3
 5.|-- 10ge7-4.core1.tor1.he.net  0.0%   100   10.1  13.9   8.9  38.6   5.3
 6.|-- tserv1.tor1.he.net         0.0%   100   13.9  10.6   9.2  17.3   1.2

mtr google.com -c 50 --report
HOST:           Loss%   Snt   Last   Avg  Best  Wrst StDev
  2.|-- grazzt-1.tunnel.tserv21.t 50.0%    50   39.1  39.2  17.4  54.6   7.2
3.|-- ge2-5.core1.tor1.he.net   44.0%    50   43.9  39.1  32.3  48.6   3.9
 4.|-- 2001:478:245:1::6         54.0%    50   37.3  38.3  26.9  72.1   8.5
 5.|-- 2001:4860::1:0:e38        48.0%    50   34.1  37.3  22.5  67.9   7.3
 6.|-- 2001:4860:0:1::71d        42.0%    50   40.7  45.7  21.6 277.8  45.1
 7.|-- yyz08s14-in-x03.1e100.net 52.0%    50   34.9  40.5  25.3 134.7  20.8

As you can see, its pretty much perfect on ipv4, but ipv6.. not so much, lol.  Thats why Im confused, Im sort of leaning to maybe something on my side, but nothing has changed, and why would it be doing 50-60% packetloss.  Maybe I should create a new tunnel to my ip from say, New York and see what happens then.

Thanks for the response.

[kasperd]

Thats why Im confused, Im sort of leaning to maybe something on my side, but nothing has changed, and why would it be doing 50-60% packetloss.

One possible explanation is that the IPv4 path treats packets differently depending on protocol. Additionally it might be that there are two parallel paths one of which can handle protocol 41 just fine and one, which drops them. Two parallel paths would explain the approximately 50% figure.

Additionally, we don't yet know in which direction the packet loss is happening.

Two ideas, that could give us additional hints about the reason. You could perform a traceroute with protocol 41 packets. With the traceroute version I have installed, the command line to do that would be: traceroute -n -P41 216.66.38.58

Additionally you could take a look on something I wrote recently on detecting in which direction drops are happening. I'd like to know, if that is useful: https://www.tunnelbroker.net/forums/index.php?topic=3069.msg18071#msg18071

Maybe I should create a new tunnel to my ip from say, New York and see what happens then.

Worth trying. But it might not help pinpoint the source of the problem. But don't you already have five tunnels?

[grazzt]

traceroute -n -P41 216.66.38.58
traceroute to 216.66.38.58 (216.66.38.58), 64 hops max, 48 byte packets
1  192.168.2.1  0.344 ms  0.312 ms  0.265 ms
2  * * *
3  * * *
4  * * *
5  * * *
6  * * *
7  * * *

done on both a freebsd machine and a linux box, always the same results.

Now I am running a tomato based router as 192.168.2.1, maybe its 'blocking' proto 41?  (the tomato router has always been there, even when ipv6 was working great for the time Ive been signed up)

I have tried forwarding proto 41 as

iptables -t nat -A WANPREROUTING -p 41 -j DNAT --to-destination lanip
iptables -A wanin -p 41 -d lanip -j ACCEPT

and it doesnt seem to change anything to the traceroutes.

I currently have 4 tunnels, so I can try another.  (seems I would have to delete my current toronto one to get a new NY one, I wont do that yet)

And your link is maybe above my head a little, did you want me to compile those programs and run them?

Thanks for the help. :-)

[kasperd]

Now I am running a tomato based router as 192.168.2.1, maybe its 'blocking' proto 41?

Where is the tunnel endpoint? If the tunnel endpoint is behind the router, then it couldn't be the router blocking the traffic, because you were after all getting some protocol 41 packets through. However do remember that since you are most likely using NAT for IPv4, you shouldn't expect to be able to have more than one computer talking protocol 41.

The traceroute command I suggested would probably give a much more useful output, if it was run on the router itself. (Assuming the router has a traceroute command with support for the -P option.)

And your link is maybe above my head a little, did you want me to compile those programs and run them?

For a start you can try to compile the client program and run it with 2a01:4f8:d16:701:9c65:a13e:5a91:4dd as the server IP address. The output from that should be useful.

You can also compile the server program and tell me the IPv6 address of the host it is running on. That doesn't have to be the router, it could be any computer on the LAN. Then I can run the client from my end to see what results it produces.

[grazzt]

The tomato router is the endpoint, it has built in support for ipv6 tunnels, etc.

Unforunately the router, while it does run linux, the traceroute command does not support passing -P option.  I suppose it might be possible to install a crosscompiler and compile a specific traceroute.  for same reason, cant run the client on the tomato router.

On two linux machines:

/client 2a01:4f8:d16:701:9c65:a13e:5a91:4dd
^C0 0 0 0  Return drop percentage estimates ?

(^C is just an incrementing number)

On a freebsd machine:

./client 2a01:4f8:d16:701:9c65:a13e:5a91:4dd
2a01:4f8:d16:701:9c65:a13e:5a91:4dd: No error: 0

Note all machines exhibit the same problem (50% packet loss)

Im also going to ask my isp about this problem.  Im using a third party cable provider, and the incumbent cable provider *loves* to play around with the third party companies.

server is running on both 2001:470:b245:0:6e62:6dff:fe8b:ffa5 (freebsd)  and 2001:470:b245:0:224:1dff:fe84:b306 (linux)

[broquea]

There were similar complaints in #ipv6 last night about this. Hopefully someone did more than complain on IRC and forums and actually opened a trouble ticket, by emailing ipv6@he.net, highlighting the IPv6 loss.

[kasperd]

On two linux machines:

/client 2a01:4f8:d16:701:9c65:a13e:5a91:4dd
^C0 0 0 0  Return drop percentage estimates ?

That means it didn't manage to get a single packet through. Looks like a three digit number, so that means you did let it run until the counter increased beyond 100?

(^C is just an incrementing number)

Some stupidity in bash or readline. Older Linux systems did not mangle the output.

On a freebsd machine:

./client 2a01:4f8:d16:701:9c65:a13e:5a91:4dd
2a01:4f8:d16:701:9c65:a13e:5a91:4dd: No error: 0

Looks like my code does not work on BSD then. Different behaviour of getaddrinfo as far as I can tell.

server is running on both 2001:470:b245:0:6e62:6dff:fe8b:ffa5 (freebsd)  and 2001:470:b245:0:224:1dff:fe84:b306 (linux)

I didn't get any packets through to either. Could it be that your router has a packet filter, which drops ICMPv6 packets of unknown type?

The server on 2a01:4f8:d16:701:9c65:a13e:5a91:4dd did not receive any packets from the client program. It did however receive ordinary echo requests from your Linux machine. Is that because you were running a ping command?

[kasperd]

There were similar complaints in #ipv6 last night about this. Hopefully someone did more than complain on IRC and forums and actually opened a trouble ticket, by emailing ipv6@he.net, highlighting the IPv6 loss.

Until your posting this thread looked like the problem might just as well be on the IPv4 path between client and server. If another user had confirmed that he saw the problem as well, I would have recommended a ticket be opened. When a service like tunnelbroker.net is provided for free, I feel obliged to do a bit more debugging before filing a ticket. I don't open a ticket before I have convinced myself, that the problem is on the HE side. But maybe that is just me.

(If I weren't limited to at most five tunnels, and at most one tunnel per client IPv4 address, I would happily configure a tunnel per tunnel server, such that I could try to reproduce problems experienced by other users, and carefully explain my findings in a ticket.)

[broquea]

It is really on the the user experiencing the issue to do the reporting, you shouldn't feel obligated to duplicate and complain on their behalf. This user could be the same that was on IRC, no idea. Tickets will garner more attention that hoping someone reads the forum that day. They are paid to read the tickets :)

[grazzt]

Unknown about the router filtering unknown ipv6 packets

Packet loss is now down to 6%, so maybe its going to fix itself??  (and now 1.7%, so maybe improving)

And once again, thanks for all the help. :-)

And no, I wasnt the one on irc complaining.  I thought I would post here first, maybe should have just opened a ticket. <shrug>.  And the problem just didnt start today, its been like this for 3 days?  I sort of waited for it to fix itself, seemed like it didnt, came here, and now it seems to be better..

sigh, and now as i try again, its at 30%. who knows. lol

[kasperd]

They are paid to read the tickets :)

Not by the users, but perhaps by somebody else. Since I am not paying for the service, I don't feel entitled to getting support from them without even looking into the problem and providing the relevant input I can provide.

[broquea]

Well shit, I should have ignored all the tickets when I ran the service then :D There were tons of tickets that never came close to providing the level of info you look up, and were wayyyyyy more demanding of better service "or their money back".

[kasperd]

Well shit, I should have ignored all the tickets when I ran the service then :D There were tons of tickets that never came close to providing the level of info you look up, and were wayyyyyy more demanding of better service "or their money back".

I won't say I am surprised. But you'll never see me suggesting users file a ticket without first collecting available information that would help pinpoint the location of the problem. Anyway, I guess the users you are referring to rarely consulted the forum before filing a ticket.

The way I see it, the forum should be useful:

• If you want help learning how to track the problem down.
• If you want advice on how to interpret the data you collected.
• If you want to know if other people are affected by the same problem.

But once you have reached the conclusion that the problem is on the HE network or getting any closer to an explanation needs information that only an administrator on the HE network could see, then you should file a ticket and not expect a forum post to reach those people at HE, who can help.