Experiencing 50% loss Europe, Amsterdam at the moment. Anyone having such issues too?
I've been seeing 20-25% packet loss at tserv4.nyc4 (New York, NY, USA) for several hours. https://tunnelbroker.net/status.php (https://tunnelbroker.net/status.php) shows green.
My traceroute [v0.94]
redacted.example.com (www.xxx.yyy.zzz) -> 209.51.161.14 2021-12-24T17:38:23-0500
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
...
7. 100ge0-35.core1.nyc7.he.net 0.0% 504 10.2 9.6 8.2 16.9 1.0
8. 100ge9-1.core1.nyc4.he.net 0.0% 504 8.4 9.2 7.8 30.2 1.8
9. tserv1.nyc4.he.net 19.2% 504 14.9 12.9 7.6 99.8 7.9
Is there any way to tell whether HE is aware of the problem, or when they believe it to be resolved?
Same regarding tserv4.nyc4.
FWIW, I've been using this tunnel for almost decade, and it's been practically flawless, excepting this week. Same issue a few days ago, but it was resolved fairly quickly. This current packet loss has been ongoing for most the day so far.
Quote from: jpwobrien on December 24, 2021, 02:43:29 PM
I've been seeing 20-25% packet loss at tserv4.nyc4 (New York, NY, USA) for several hours. https://tunnelbroker.net/status.php (https://tunnelbroker.net/status.php) shows green.
Same here. Still going after 13 hours.
I am also having this issue with tserv4.nyc4. It went down to 0% loss overnight (Eastern) but the problem is back now.
I've emailed ipv6@he.net about the issue since the server status is still showing okay.
Quote from: noahajac on December 25, 2021, 05:34:18 AM
I am also having this issue with tserv4.nyc4. It went down to 0% loss overnight (Eastern) but the problem is back now.
I've emailed ipv6@he.net about the issue since the server status is still showing okay.
My logs show the same thing. Between about 1am and 7am Eastern time, things looked ok with no loss, but as of 7am it's doing the same thing.
I've also sent an email to support@he.net.
Same issue here from Philadelphia area to tserv1.nyc4.he.net.
Showing about 18% loss on IPv4 from Verizon and Comcast in Philly.
Hops look good all the way to 100ge1-1.core1.nyc4.he.net but then it's bad for last hop.
Didn't notice it until this morning.
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. gw-01-365.hslc.org 0.0% 880 0.3 0.2 0.2 1.5 0.0
2. 66.216.50.65 0.0% 880 1.5 1.4 1.2 6.8 0.2
3. be4.ash-cr01.365datacenters.com 0.0% 880 4.9 4.7 4.5 12.8 0.4
4. as6939.10gigabitethernet2-2.core1.ash1.he.net 0.0% 880 4.1 7.1 4.0 287.1 12.3
5. 100ge1-1.core1.nyc4.he.net 0.0% 879 6.8 9.7 6.6 71.9 7.5
6. tserv1.nyc4.he.net 18.1% 879 6.7 11.0 6.5 110.1 10.8
Thanks all. I know I am not the only one who sees / has these IPv6 connectivity issues at the moment.
Quote from: noahajac on December 25, 2021, 05:34:18 AM
I am also having this issue with tserv4.nyc4. It went down to 0% loss overnight (Eastern) but the problem is back now.
I've emailed ipv6@he.net about the issue since the server status is still showing okay.
Thanks. I did just send HE an e-mail too.
I did include some ping6 tests from a VPS which is located in a data center.
Besides these results I did also include screenshots of my time server. The difference between IPv4 and IPv6 to the same server are huge at the moment.
(https://ict-diensten.com/pub/img/stats-ntp.png)
https://ict-diensten.com/pub/img/stats-ntp.png (https://ict-diensten.com/pub/img/stats-ntp.png)
(https://ict-diensten.com/pub/img/stats-ntp-ipv4.png)
https://ict-diensten.com/pub/img/stats-ntp-ipv4.png (https://ict-diensten.com/pub/img/stats-ntp-ipv4.png)
It seems to have settled down, not seeing any loss at this point so I have re-enabled my tunnel.
Great work fixing it up on Christmas day!
Unfortunately still experiencing issues.
I did receive the following reply
Hello Bouke,
The IP 2001:470:1f14:71::1 belongs to our core infrastructure, and is not intended to be used as a dedicated endpoint for measuring latency or uptime. ICMP responses are deprioritized in favor of routing traffic. This depriotization can lead to artificially high packet loss and/or latency.
If you are still experiencing issues, please provide a MTR with at least 100 samples, and if possible, also in the reverse direction.
[Redacted: removed name]
Hurricane Electric Internet Services
510-580-4100
I just did disable gateway monitoring.
Not sure why I have to provide at least 100 samples? That's quite a lot to point out a problem.
#
mtr --report 2001:4860:4860::8888
Start: 2021-12-26T12:01:48+0100
HOST: s2 Loss% Snt Last Avg Best Wrst StDev
1.|-- 2001:470:1f15:73::1 0.0% 10 0.3 0.4 0.3 0.5 0.1
2.|-- tunnel479523.tunnel.tserv 90.0% 10 11.3 11.3 11.3 11.3 0.0
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- amsix-router.google.com 0.0% 10 9.9 10.5 9.2 14.3 2.0
5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
6.|-- 2001:4860:0:1::518f 0.0% 10 9.3 9.4 8.9 10.0 0.3
7.|-- dns.google 50.0% 10 10.4 16.6 10.4 40.9 13.6
# mtr --report 2620:fe::fe
Start: 2021-12-26T12:09:05+0100
HOST: s2 Loss% Snt Last Avg Best Wrst StDev
1.|-- 2001:470:1f15:73::1 0.0% 10 0.5 0.4 0.4 0.5 0.0
2.|-- tunnel479523.tunnel.tserv 0.0% 10 10.4 12.0 9.5 26.2 5.0
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
5.|-- dns.quad9.net 50.0% 10 8.1 8.1 8.0 8.3 0.1
# mtr -6 --report s2.gigabitjes.nl
Start: 2021-12-26T11:11:24+0000
HOST: s3 Loss% Snt Last Avg Best Wrst StDev
1.|-- 2a01:7c8:bb09::2 0.0% 10 0.8 17.8 0.8 89.4 31.6
2.|-- l15-f1.ams4.transip.net 0.0% 10 9.6 7.2 1.7 10.6 3.1
3.|-- s1.r2.ams0.transip.net 0.0% 10 0.5 1.0 0.5 2.6 0.8
4.|-- r2-a0.e1.ams8.transip.net 0.0% 10 0.6 0.6 0.5 0.7 0.1
5.|-- asd-s5-rou-1041.NL.as286. 0.0% 10 0.7 0.9 0.7 1.9 0.4
6.|-- 2001:668:0:2:ffff:0:5995: 0.0% 10 31.9 9.1 6.3 31.9 8.0
7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
8.|-- e0-31.core2.lon3.he.net 70.0% 10 8.1 8.0 7.8 8.1 0.2
9.|-- hurricane-electric.nikhef 40.0% 10 7.4 7.5 7.3 7.7 0.1
10.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
11.|-- tserv1.ams1.he.net 0.0% 10 8.1 12.7 7.8 53.1 14.2
12.|-- tunnel479523-pt.tunnel.ts 60.0% 10 13.9 13.9 13.6 14.2 0.2
13.|-- s2.gigabitjes.nl 40.0% 10 15.7 14.7 13.8 17.0 1.3
# mtr -6 --report time.s9.gigabitjes.nl
Start: 2021-12-26T11:12:31+0000
HOST: s3 Loss% Snt Last Avg Best Wrst StDev
1.|-- 2a01:7c8:bb09::2 0.0% 10 0.8 3.9 0.7 31.8 9.8
2.|-- l15-f2.ams4.transip.net 0.0% 10 7.9 6.1 0.7 10.0 3.1
3.|-- s2.r1.ams0.transip.net 0.0% 10 0.5 2.1 0.4 15.8 4.8
4.|-- r1-a0.e1.ams7.transip.net 0.0% 10 0.6 0.6 0.5 0.7 0.1
5.|-- xe-1-5-1-0.a01.amstnl07.n 0.0% 10 1.2 3.0 1.1 9.5 2.8
6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
7.|-- tserv1.ams1.he.net 0.0% 10 2.8 2.8 2.5 3.0 0.2
8.|-- tunnel479523-pt.tunnel.ts 50.0% 10 9.3 9.1 9.0 9.3 0.2
9.|-- 2001:470:1f15:73::1000:3f 30.0% 10 15.0 12.4 9.6 22.4 4.8
Does someone have some ideas? Thanks.
I will kindly reply to the e-mail and ask what's the best way to gather more data. Maybe I could set something up to continuously run mtr both ways.
same issue here, although I've had no response from he.net on my issue
user@Mac-mini ~ % sudo mtr -6 --report 2620:fe::fe
Start: 2021-12-26T14:04:24+0100
HOST: Mac-mini.local Loss% Snt Last Avg Best Wrst StDev
1.|-- 2001:470:xxxx:xxxx::2 0.0% 10 0.7 0.8 0.7 0.9 0.0
2.|-- tunnel689537.tunnel.tserv 90.0% 10 16.7 16.7 16.7 16.7 0.0
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
5.|-- 2620:fe::fe 60.0% 10 14.9 14.9 14.8 15.1 0.1
and
user@Mac-mini ~ % sudo mtr -6 --report tweakers.net
Start: 2021-12-26T14:03:25+0100
HOST: Mac-mini.local Loss% Snt Last Avg Best Wrst StDev
1.|-- 2001:470:xxxx:xxxx::2 0.0% 10 0.6 0.7 0.6 0.8 0.1
2.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- 2001:7f8:1::a501:5703:1 0.0% 10 23.6 14.6 13.2 23.6 3.2
5.|-- 2001:9a8:0:78::1 0.0% 10 14.3 19.9 13.5 44.5 10.1
6.|-- 2001:9a8:0:78::14 90.0% 10 16.3 16.3 16.3 16.3 0.0
7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
8.|-- tweakers.net 50.0% 10 41.0 19.1 13.3 41.0 12.3
Maybe some peering issue?
I did some tinkering with mtr and came up with this:
mtr -6 -rw -c 500 time.s9.gigabitjes.nl >/tmp/s3-ipv6-time-s9.log
mtr -6 -rw -c 500 s3.gigabitjes.nl > /tmp/time-s9-ipv6-s3.log
This will record 500 pings from a vps to my ntp server at home and also from my ntp server to the vps.
The VPS is located in the network of TransIP and the server at home is a vm in Proxmox.
I am hoping HE wishes to see something like that?
Hi ICTD01, got this for you from my end
Mac-mini.local (2001:470:1f15:33b:19f3:34e9:837a:5778) -> tweakers.net 2021-12-26T15:04:00+0100
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2001:470:xxxx:xxxx::2 0.0% 100 0.7 0.6 0.5 1.3 0.1
2. 2001:470:xxxx:xxxx::1 0.0% 100 16.5 17.6 15.0 46.5 5.1
3. (waiting for reply)
4. 2001:7f8:1::a501:5703:1 0.0% 100 37.8 15.1 12.7 43.2 5.1
5. 2001:9a8:0:78::1 1.0% 100 13.6 27.0 13.4 125.5 25.1
6. 2001:9a8:0:78::14 99.0% 100 14.8 14.8 14.8 14.8 0.0
7. (waiting for reply)
8. 2001:9a8:0:e:1337:0:80:1 61.6% 100 13.3 15.4 12.8 55.6 8.8
Mac-mini.local (2001:470:1f15:33b:19f3:34e9:837a:5778) -> ipv6.google.com 2021-12-26T15:04:14+0100
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2001:470:xxxx:xxxx::2 0.0% 100 0.9 0.7 0.5 1.8 0.2
2. (waiting for reply)
3. (waiting for reply)
4. 2001:7f8:1::a501:5169:1 0.0% 100 14.6 16.5 13.9 59.6 6.4
5. 2001:4860:0:f8b::c 15.0% 100 14.1 15.7 13.8 48.2 5.8
6. 2001:4860::c:4002:54bb 99.0% 100 26.4 26.4 26.4 26.4 0.0
7. 2001:4860::c:4002:365b 0.0% 100 25.6 27.8 25.3 64.1 6.8
8. 2001:4860::1:0:d0d8 58.6% 100 26.5 28.6 26.5 55.1 5.0
9. 2001:4860:0:1::4933 99.0% 100 26.9 26.9 26.9 26.9 0.0
10. 2a00:1450:4001:803::200e 44.0% 100 24.9 25.7 23.9 58.5 5.6
Hello wijgaart,
I did run a test targeting tweakers.net too and the results indicate significant loss. This test is from my home network - IPv6 connectivity via HE.
# mtr -6 -rw -c 100 tweakers.net >/tmp/t1-ipv6-tweakers
# cat /tmp/t1-ipv6-tweakers
Start: 2021-12-26T20:38:06+0100
HOST: t1 Loss% Snt Last Avg Best Wrst StDev
1.|-- 2001:470:1f15:73::1 0.0% 100 0.5 0.5 0.4 0.6 0.0
2.|-- tunnel479523.tunnel.tserv11.ams1.ipv6.he.net 0.0% 100 12.4 12.3 10.6 37.4 2.9
3.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
4.|-- 2001:7f8:1::a501:5703:1 0.0% 100 10.9 12.8 10.1 44.2 5.6
5.|-- 2001:9a8:0:78::3 0.0% 100 97.1 20.1 10.1 97.1 18.9
6.|-- 2001:9a8:0:78::16 0.0% 100 10.6 17.1 10.2 101.2 15.0
7.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
8.|-- tweakers.net 48.0% 100 9.5 10.8 9.0 56.3 7.0
When I run the same test from a VPS (which is located at TransIP, Rotterdam), then the result is fine.
That doesn't have anything to do with HE - other than that it is a good way to shows the difference between problematic IPv6 connectivity and good IPv6 connectivity.
# cat /tmp/t1-ipv6-tweakers
Start: 2021-12-26T20:56:30+0100
HOST: s1 Loss% Snt Last Avg Best Wrst StDev
1.|-- gw.rtm0.transip.net 0.0% 100 11.2 7.6 0.6 92.8 10.1
2.|-- 2a01:7c8:d00f:20::1 0.0% 100 5.2 6.5 0.5 38.4 4.8
3.|-- 2a01:7c8:d00f:41::1 0.0% 100 1.2 0.9 0.4 19.9 2.1
4.|-- r1.rtm0-ae24.router1.ams0.transip.net 0.0% 100 1.5 1.8 1.4 20.5 2.5
5.|-- r1-a0.e1.ams0.transip.net 0.0% 100 2.5 3.5 2.1 38.1 5.6
6.|-- 2001:7f8:1::a501:5703:1 0.0% 100 2.2 3.4 1.9 33.4 3.9
7.|-- 2001:9a8:0:78::1 0.0% 100 3.0 14.8 2.8 116.1 26.6
8.|-- 2001:9a8:0:78::14 0.0% 100 2.4 9.7 2.2 105.4 16.0
9.|-- e600-1.eun-leaf-04.true.nl 0.0% 100 4.7 8.7 3.0 21.4 4.7
10.|-- tweakers.net 0.0% 100 2.1 2.0 1.9 2.4 0.1
I'd say that this is enough to prove that HE engineers need to investigate.
I haven't heard from them yet though.
Got this
Please retest and advise if you still see an issue.
Thanks,
Gene
HE Support
Seems they found some issue, without disclosing what was wrong. Whatever it was it solved the issue for me
Hi Gene,
Whatever you have done it seems to have fixed the issue. As a fellow network administrator Im curious what the issue was and how you have resolved it.
Thanks for your effort,
Cheers,
Michiel
They answered
There was a circuit with some odd loss in it we found in AMS, and once we took it out of a bundle, the problems were solved.
Thanks,
Gene
HE Support
So I guess case closed, I'm curious if the problem has been resolved for you too ictd01?
Quote from: wijgaart on December 26, 2021, 10:29:01 PM
Got this
Please retest and advise if you still see an issue.
Thanks,
Gene
HE Support
Seems they found some issue, without disclosing what was wrong. Whatever it was it solved the issue for me
Yes. I did receive an email from Gene too - and it's fixed.
happy days :)