50% loss Europe, Amsterdam

ictd01 · December 24, 2021, 09:49:09 AM

Experiencing 50% loss Europe, Amsterdam at the moment. Anyone having such issues too?

jpwobrien · December 24, 2021, 02:43:29 PM

I've been seeing 20-25% packet loss at tserv4.nyc4 (New York, NY, USA) for several hours. https://tunnelbroker.net/status.php shows green.

Code Select

                               My traceroute  [v0.94]
redacted.example.com (www.xxx.yyy.zzz) -> 209.51.161.14    2021-12-24T17:38:23-0500
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                           Packets               Pings
 Host                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
...
 7. 100ge0-35.core1.nyc7.he.net           0.0%   504   10.2   9.6   8.2  16.9   1.0
 8. 100ge9-1.core1.nyc4.he.net            0.0%   504    8.4   9.2   7.8  30.2   1.8
 9. tserv1.nyc4.he.net                   19.2%   504   14.9  12.9   7.6  99.8   7.9

Is there any way to tell whether HE is aware of the problem, or when they believe it to be resolved?

ziddey · December 24, 2021, 03:56:36 PM

Same regarding tserv4.nyc4.

FWIW, I've been using this tunnel for almost decade, and it's been practically flawless, excepting this week. Same issue a few days ago, but it was resolved fairly quickly. This current packet loss has been ongoing for most the day so far.

steve1515 · December 24, 2021, 09:29:54 PM

Quote from: jpwobrien on December 24, 2021, 02:43:29 PM
I've been seeing 20-25% packet loss at tserv4.nyc4 (New York, NY, USA) for several hours. https://tunnelbroker.net/status.php shows green.

Same here. Still going after 13 hours.

noahajac · December 25, 2021, 05:34:18 AM

I am also having this issue with tserv4.nyc4. It went down to 0% loss overnight (Eastern) but the problem is back now.

I've emailed ipv6@he.net about the issue since the server status is still showing okay.

steve1515 · December 25, 2021, 07:54:24 AM

Quote from: noahajac on December 25, 2021, 05:34:18 AM
I am also having this issue with tserv4.nyc4. It went down to 0% loss overnight (Eastern) but the problem is back now.

I've emailed ipv6@he.net about the issue since the server status is still showing okay.

My logs show the same thing. Between about 1am and 7am Eastern time, things looked ok with no loss, but as of 7am it's doing the same thing.

I've also sent an email to support@he.net.

gcarey3 · December 25, 2021, 09:04:14 AM

Same issue here from Philadelphia area to tserv1.nyc4.he.net.

Showing about 18% loss on IPv4 from Verizon and Comcast in Philly.

Hops look good all the way to 100ge1-1.core1.nyc4.he.net but then it's bad for last hop.

Didn't notice it until this morning.

Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. gw-01-365.hslc.org 0.0% 880 0.3 0.2 0.2 1.5 0.0
2. 66.216.50.65 0.0% 880 1.5 1.4 1.2 6.8 0.2
3. be4.ash-cr01.365datacenters.com 0.0% 880 4.9 4.7 4.5 12.8 0.4
4. as6939.10gigabitethernet2-2.core1.ash1.he.net 0.0% 880 4.1 7.1 4.0 287.1 12.3
5. 100ge1-1.core1.nyc4.he.net 0.0% 879 6.8 9.7 6.6 71.9 7.5
6. tserv1.nyc4.he.net 18.1% 879 6.7 11.0 6.5 110.1 10.8

ictd01 · December 25, 2021, 01:49:12 PM

Thanks all. I know I am not the only one who sees / has these IPv6 connectivity issues at the moment.

Quote from: noahajac on December 25, 2021, 05:34:18 AM
I am also having this issue with tserv4.nyc4. It went down to 0% loss overnight (Eastern) but the problem is back now.

I've emailed ipv6@he.net about the issue since the server status is still showing okay.

Thanks. I did just send HE an e-mail too.

I did include some ping6 tests from a VPS which is located in a data center.

Besides these results I did also include screenshots of my time server. The difference between IPv4 and IPv6 to the same server are huge at the moment.

https://ict-diensten.com/pub/img/stats-ntp.png

https://ict-diensten.com/pub/img/stats-ntp-ipv4.png

gcarey3 · December 25, 2021, 05:33:54 PM

It seems to have settled down, not seeing any loss at this point so I have re-enabled my tunnel.

Great work fixing it up on Christmas day!

ictd01 · December 26, 2021, 03:14:30 AM

Unfortunately still experiencing issues.

I did receive the following reply

Code Select

Hello Bouke,

The IP 2001:470:1f14:71::1 belongs to our core infrastructure, and is not intended to be used as a dedicated endpoint for measuring latency or uptime. ICMP responses are deprioritized in favor of routing traffic. This depriotization can lead to artificially high packet loss and/or latency.

If you are still experiencing issues, please provide a MTR with at least 100 samples, and if possible, also in the reverse direction.

[Redacted: removed name]
Hurricane Electric Internet Services
510-580-4100

I just did disable gateway monitoring.

Not sure why I have to provide at least 100 samples? That's quite a lot to point out a problem.

Code Select

# 
mtr --report 2001:4860:4860::8888
Start: 2021-12-26T12:01:48+0100
HOST: s2                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:470:1f15:73::1        0.0%    10    0.3   0.4   0.3   0.5   0.1
  2.|-- tunnel479523.tunnel.tserv 90.0%    10   11.3  11.3  11.3  11.3   0.0
  3.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- amsix-router.google.com    0.0%    10    9.9  10.5   9.2  14.3   2.0
  5.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  6.|-- 2001:4860:0:1::518f        0.0%    10    9.3   9.4   8.9  10.0   0.3
  7.|-- dns.google                50.0%    10   10.4  16.6  10.4  40.9  13.6

# mtr --report 2620:fe::fe         
Start: 2021-12-26T12:09:05+0100
HOST: s2                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:470:1f15:73::1        0.0%    10    0.5   0.4   0.4   0.5   0.0
  2.|-- tunnel479523.tunnel.tserv  0.0%    10   10.4  12.0   9.5  26.2   5.0
  3.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- dns.quad9.net             50.0%    10    8.1   8.1   8.0   8.3   0.1

# mtr -6 --report s2.gigabitjes.nl
Start: 2021-12-26T11:11:24+0000
HOST: s3                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2a01:7c8:bb09::2           0.0%    10    0.8  17.8   0.8  89.4  31.6
  2.|-- l15-f1.ams4.transip.net    0.0%    10    9.6   7.2   1.7  10.6   3.1
  3.|-- s1.r2.ams0.transip.net     0.0%    10    0.5   1.0   0.5   2.6   0.8
  4.|-- r2-a0.e1.ams8.transip.net  0.0%    10    0.6   0.6   0.5   0.7   0.1
  5.|-- asd-s5-rou-1041.NL.as286.  0.0%    10    0.7   0.9   0.7   1.9   0.4
  6.|-- 2001:668:0:2:ffff:0:5995:  0.0%    10   31.9   9.1   6.3  31.9   8.0
  7.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  8.|-- e0-31.core2.lon3.he.net   70.0%    10    8.1   8.0   7.8   8.1   0.2
  9.|-- hurricane-electric.nikhef 40.0%    10    7.4   7.5   7.3   7.7   0.1
 10.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 11.|-- tserv1.ams1.he.net         0.0%    10    8.1  12.7   7.8  53.1  14.2
 12.|-- tunnel479523-pt.tunnel.ts 60.0%    10   13.9  13.9  13.6  14.2   0.2
 13.|-- s2.gigabitjes.nl          40.0%    10   15.7  14.7  13.8  17.0   1.3

# mtr -6 --report time.s9.gigabitjes.nl
Start: 2021-12-26T11:12:31+0000
HOST: s3                          Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2a01:7c8:bb09::2           0.0%    10    0.8   3.9   0.7  31.8   9.8
  2.|-- l15-f2.ams4.transip.net    0.0%    10    7.9   6.1   0.7  10.0   3.1
  3.|-- s2.r1.ams0.transip.net     0.0%    10    0.5   2.1   0.4  15.8   4.8
  4.|-- r1-a0.e1.ams7.transip.net  0.0%    10    0.6   0.6   0.5   0.7   0.1
  5.|-- xe-1-5-1-0.a01.amstnl07.n  0.0%    10    1.2   3.0   1.1   9.5   2.8
  6.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  7.|-- tserv1.ams1.he.net         0.0%    10    2.8   2.8   2.5   3.0   0.2
  8.|-- tunnel479523-pt.tunnel.ts 50.0%    10    9.3   9.1   9.0   9.3   0.2
  9.|-- 2001:470:1f15:73::1000:3f 30.0%    10   15.0  12.4   9.6  22.4   4.8

Does someone have some ideas? Thanks.

I will kindly reply to the e-mail and ask what's the best way to gather more data. Maybe I could set something up to continuously run mtr both ways.

wijgaart · December 26, 2021, 05:11:40 AM

same issue here, although I've had no response from he.net on my issue

Code Select


user@Mac-mini ~ % sudo mtr -6 --report 2620:fe::fe
Start: 2021-12-26T14:04:24+0100
HOST: Mac-mini.local  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:470:xxxx:xxxx::2       0.0%    10    0.7   0.8   0.7   0.9   0.0
  2.|-- tunnel689537.tunnel.tserv 90.0%    10   16.7  16.7  16.7  16.7   0.0
  3.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  5.|-- 2620:fe::fe               60.0%    10   14.9  14.9  14.8  15.1   0.1

and

user@Mac-mini ~ % sudo mtr -6 --report tweakers.net
Start: 2021-12-26T14:03:25+0100
HOST: Mac-mini.local  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:470:xxxx:xxxx::2       0.0%    10    0.6   0.7   0.6   0.8   0.1
  2.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- 2001:7f8:1::a501:5703:1    0.0%    10   23.6  14.6  13.2  23.6   3.2
  5.|-- 2001:9a8:0:78::1           0.0%    10   14.3  19.9  13.5  44.5  10.1
  6.|-- 2001:9a8:0:78::14         90.0%    10   16.3  16.3  16.3  16.3   0.0
  7.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  8.|-- tweakers.net              50.0%    10   41.0  19.1  13.3  41.0  12.3

Maybe some peering issue?

ictd01 · December 26, 2021, 05:28:41 AM

I did some tinkering with mtr and came up with this:

mtr -6 -rw -c 500 time.s9.gigabitjes.nl >/tmp/s3-ipv6-time-s9.log
mtr -6 -rw -c 500 s3.gigabitjes.nl > /tmp/time-s9-ipv6-s3.log

This will record 500 pings from a vps to my ntp server at home and also from my ntp server to the vps.
The VPS is located in the network of TransIP and the server at home is a vm in Proxmox.

I am hoping HE wishes to see something like that?

wijgaart · December 26, 2021, 06:06:38 AM

Hi ICTD01, got this for you from my end

Code Select


Mac-mini.local (2001:470:1f15:33b:19f3:34e9:837a:5778) -> tweakers.net                         2021-12-26T15:04:00+0100
                                                                                           Packets               Pings
 Host                                                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 2001:470:xxxx:xxxx::2                                                                  0.0%   100    0.7   0.6   0.5   1.3   0.1
 2. 2001:470:xxxx:xxxx::1                                                                  0.0%   100   16.5  17.6  15.0  46.5   5.1
 3. (waiting for reply)
 4. 2001:7f8:1::a501:5703:1                                                               0.0%   100   37.8  15.1  12.7  43.2   5.1
 5. 2001:9a8:0:78::1                                                                      1.0%   100   13.6  27.0  13.4 125.5  25.1
 6. 2001:9a8:0:78::14                                                                    99.0%   100   14.8  14.8  14.8  14.8   0.0
 7. (waiting for reply)
 8. 2001:9a8:0:e:1337:0:80:1                                                             61.6%   100   13.3  15.4  12.8  55.6   8.8

Mac-mini.local (2001:470:1f15:33b:19f3:34e9:837a:5778) -> ipv6.google.com                      2021-12-26T15:04:14+0100
                                                                                           Packets               Pings
 Host                                                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 2001:470:xxxx:xxxx::2                                                                  0.0%   100    0.9   0.7   0.5   1.8   0.2
 2. (waiting for reply)
 3. (waiting for reply)
 4. 2001:7f8:1::a501:5169:1                                                               0.0%   100   14.6  16.5  13.9  59.6   6.4
 5. 2001:4860:0:f8b::c                                                                   15.0%   100   14.1  15.7  13.8  48.2   5.8
 6. 2001:4860::c:4002:54bb                                                               99.0%   100   26.4  26.4  26.4  26.4   0.0
 7. 2001:4860::c:4002:365b                                                                0.0%   100   25.6  27.8  25.3  64.1   6.8
 8. 2001:4860::1:0:d0d8                                                                  58.6%   100   26.5  28.6  26.5  55.1   5.0
 9. 2001:4860:0:1::4933                                                                  99.0%   100   26.9  26.9  26.9  26.9   0.0
10. 2a00:1450:4001:803::200e                                                             44.0%   100   24.9  25.7  23.9  58.5   5.6

ictd01 · December 26, 2021, 12:01:48 PM

Hello wijgaart,

I did run a test targeting tweakers.net too and the results indicate significant loss. This test is from my home network - IPv6 connectivity via HE.

Code Select


# mtr -6 -rw -c 100 tweakers.net >/tmp/t1-ipv6-tweakers  

# cat /tmp/t1-ipv6-tweakers
Start: 2021-12-26T20:38:06+0100
HOST: t1                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:470:1f15:73::1                           0.0%   100    0.5   0.5   0.4   0.6   0.0
  2.|-- tunnel479523.tunnel.tserv11.ams1.ipv6.he.net  0.0%   100   12.4  12.3  10.6  37.4   2.9
  3.|-- ???                                          100.0   100    0.0   0.0   0.0   0.0   0.0
  4.|-- 2001:7f8:1::a501:5703:1                       0.0%   100   10.9  12.8  10.1  44.2   5.6
  5.|-- 2001:9a8:0:78::3                              0.0%   100   97.1  20.1  10.1  97.1  18.9
  6.|-- 2001:9a8:0:78::16                             0.0%   100   10.6  17.1  10.2 101.2  15.0
  7.|-- ???                                          100.0   100    0.0   0.0   0.0   0.0   0.0
  8.|-- tweakers.net                                 48.0%   100    9.5  10.8   9.0  56.3   7.0

When I run the same test from a VPS (which is located at TransIP, Rotterdam), then the result is fine.
That doesn't have anything to do with HE - other than that it is a good way to shows the difference between problematic IPv6 connectivity and good IPv6 connectivity.

Code Select


# cat /tmp/t1-ipv6-tweakers
Start: 2021-12-26T20:56:30+0100
HOST: s1                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- gw.rtm0.transip.net                    0.0%   100   11.2   7.6   0.6  92.8  10.1
  2.|-- 2a01:7c8:d00f:20::1                    0.0%   100    5.2   6.5   0.5  38.4   4.8
  3.|-- 2a01:7c8:d00f:41::1                    0.0%   100    1.2   0.9   0.4  19.9   2.1
  4.|-- r1.rtm0-ae24.router1.ams0.transip.net  0.0%   100    1.5   1.8   1.4  20.5   2.5
  5.|-- r1-a0.e1.ams0.transip.net              0.0%   100    2.5   3.5   2.1  38.1   5.6
  6.|-- 2001:7f8:1::a501:5703:1                0.0%   100    2.2   3.4   1.9  33.4   3.9
  7.|-- 2001:9a8:0:78::1                       0.0%   100    3.0  14.8   2.8 116.1  26.6
  8.|-- 2001:9a8:0:78::14                      0.0%   100    2.4   9.7   2.2 105.4  16.0
  9.|-- e600-1.eun-leaf-04.true.nl             0.0%   100    4.7   8.7   3.0  21.4   4.7
 10.|-- tweakers.net                           0.0%   100    2.1   2.0   1.9   2.4   0.1

wijgaart · December 26, 2021, 01:02:46 PM

I'd say that this is enough to prove that HE engineers need to investigate.

I haven't heard from them yet though.

News:

50% loss Europe, Amsterdam