Packet Loss on tserv4.nyc4

[noahajac]

Hello all,

I have a tunnel on tserv4.nyc4 and, over the past couple of weeks, have encountered a noticeable decrease in connection quality. This seems to manifest especially when establishing HTTPS connections, causing slow loading times or sometimes timeouts. This only occurs on IPv6 sites.

Doing some testing, I have discovered intermittent packet loss when pinging the tunnel endpoint, 209.51.161.14, and as a result am also seeing packet loss when accessing IPv6 addresses.

I have reached out to ipv6@he.net and have sent over mtr results, but have not yet received a response. In the meantime I figured I'd post here to see if anyone else is experiencing this issue.

Thank you!

[esaym]

Similar issue here. Have had poor ipv6 performance for nearly two weeks now. Finally got around to looking into the matter. Both the provided ipv4 and ipv6 endpoint gateways for my tunnel will get random bouts of 50% packet loss when pinging them. It will resolve itself after a minute or so and then 5 to 10 minutes later will happen again. Today was the worst, hence looking into the matter. I am using tserv8.dal1 which is close to me. Strangely tserv1.den1 which is the next closest to me doesn't seem to get these ping spikes. But I can't change my endpoints without tearing down the tunnel and creating a fresh one right? I don't want to have to deal with changing my prefix and redoing DNS.

[papamidnight]

You are not the only one, and I've seen various other threads on this board reporting similar issues pertaining to IPv6 tunnels. In example, there are these threads:

• https://forums.he.net/index.php?topic=4249.0
• https://forums.he.net/index.php?topic=4251.0
• https://forums.he.net/index.php?topic=4237.0

People seem to be seeing varying behaviors ranging from high latency, to packet loss, to dropped and out-of-order packets, among other issues.

For myself, I've had a tunnel with the Asbhurn, VA endpoint (216.66.22.2) configured for the better part of 10 years now. I had to unfortunately disable my tunnel as, beginning in April of 2023, I started experiencing significant amounts of packet loss to IPv6 addresses - though none to the endpoint itself via IPv4. It was becoming exceptionally detrimental where some applications would fail to work as they were attempting to connect after defaulting to IPv6 first. Browsers would fall back to IPv4, but after quite an extended time period.

It's so bad that I literally can't even do a speedtest using the Ookla CLI client:

Code Select Expand

~$ speedtest                                                                                                 [40/40]
[2023-05-30 02:55:25.155] [error] Configuration - Timeout was reached (TimeoutException)
[2023-05-30 02:55:25.156] [error] Configuration - Cannot retrieve configuration document (0)
[2023-05-30 02:55:25.172] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[2023-05-30 02:55:25.172] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[error] Configuration - Could not retrieve or read configuration (ConfigurationError)

~$ speedtest --servers
Closest servers:

    ID  Name                           Location             Country
==============================================================================
  1775  Comcast                        Baltimore, MD        United States
  8987  Port Networks                  Baltimore, MD        United States
 30823  ETTE                           Washington, DC       United States
 21542  i3D.net                        Ashburn, VA          United States
 16617  Cox - Nova                     Fairfax, VA          United States
 14229  Frontier                       Ashburn, VA          United States
 17383  Windstream                     Ashburn, VA          United States
 21640  Pilot Fiber                    Ashburn, VA          United States
 23373  Whitesky Communications LLC    Ashburn, VA          United States
 37568  Netprotect                     Ashburn, VA          United States
~$ speedtest --server-id=1775
[2023-05-30 02:57:41.016] [error] Configuration - Timeout was reached (TimeoutException)
[2023-05-30 02:57:41.017] [error] Configuration - Cannot retrieve configuration document (0)
[2023-05-30 02:57:41.023] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[2023-05-30 02:57:41.023] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[error] Configuration - Could not retrieve or read configuration (ConfigurationError)

I thought about creating a new tunnel to see what the behavior might be on another endpoint, but then I saw these threads so I held off on that. The next thing I'm trying is what was in this thread on reddit - as what they are describing therein is exactly what I've been experiencing.

As an aside, I'm hoping that someone at HE reads the report that you sent in via email, or this or any of the other threads regarding any of the problems pertaining to the tunnel service, and is able to determine a cause and provide a resolution.

Edit: That did not work. I am trying a different set of settings based on this thread on the Ubiquiti board.

Edit 2: I made the following changes and re-enabled my tunnel GIF interface. So far, the follow settings for me seem to have had a positive effect:

• Tunnel MTU: 1480
• Interface MTU: 1480
• Interface MSS: 1420

[kamil445]

From 4/2023 to 5/2023 there are 11001 new accounts!

HE tried ban spam/bot accounts 1/2023,2/2023,3/2023,4/2023 (or inactive?) but looks like they still creating new accounts.

Other tunnelbroker (route48) has been shut down because of abuse, most of abuse was from China (they tried to bypass government firewall to access youtube, facebook etc).

route48 is gone, so they probably now trying Hurricane tunnels, because of that, probably we have these problems here....

Also in Europe we have a war, so maybe spammers use tunnel to spread fake news about war or just try to overload network infrastructure (?).

Or just stupid kids DDOS-ing HE endpoints for fun..

This is sad, because we can't do something, we need just wait for Hurricane staff, but please don't shut down this great service like route48 did.

[noahajac]

You are not the only one, and I've seen various other threads on this board reporting similar issues pertaining to IPv6 tunnels. In example, there are these threads:

• https://forums.he.net/index.php?topic=4249.0
• https://forums.he.net/index.php?topic=4251.0
• https://forums.he.net/index.php?topic=4237.0

People seem to be seeing varying behaviors ranging from high latency, to packet loss, to dropped and out-of-order packets, among other issues.

For myself, I've had a tunnel with the Asbhurn, VA endpoint (216.66.22.2) configured for the better part of 10 years now. I had to unfortunately disable my tunnel as, beginning in April of 2023, I started experiencing significant amounts of packet loss to IPv6 addresses - though none to the endpoint itself via IPv4. It was becoming exceptionally detrimental where some applications would fail to work as they were attempting to connect after defaulting to IPv6 first. Browsers would fall back to IPv4, but after quite an extended time period.

It's so bad that I literally can't even do a speedtest using the Ookla CLI client:

Code Select Expand

~$ speedtest                                                                                                 [40/40]
[2023-05-30 02:55:25.155] [error] Configuration - Timeout was reached (TimeoutException)
[2023-05-30 02:55:25.156] [error] Configuration - Cannot retrieve configuration document (0)
[2023-05-30 02:55:25.172] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[2023-05-30 02:55:25.172] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[error] Configuration - Could not retrieve or read configuration (ConfigurationError)

~$ speedtest --servers
Closest servers:

    ID  Name                           Location             Country
==============================================================================
  1775  Comcast                        Baltimore, MD        United States
  8987  Port Networks                  Baltimore, MD        United States
 30823  ETTE                           Washington, DC       United States
 21542  i3D.net                        Ashburn, VA          United States
 16617  Cox - Nova                     Fairfax, VA          United States
 14229  Frontier                       Ashburn, VA          United States
 17383  Windstream                     Ashburn, VA          United States
 21640  Pilot Fiber                    Ashburn, VA          United States
 23373  Whitesky Communications LLC    Ashburn, VA          United States
 37568  Netprotect                     Ashburn, VA          United States
~$ speedtest --server-id=1775
[2023-05-30 02:57:41.016] [error] Configuration - Timeout was reached (TimeoutException)
[2023-05-30 02:57:41.017] [error] Configuration - Cannot retrieve configuration document (0)
[2023-05-30 02:57:41.023] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[2023-05-30 02:57:41.023] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[error] Configuration - Could not retrieve or read configuration (ConfigurationError)

I thought about creating a new tunnel to see what the behavior might be on another endpoint, but then I saw these threads so I held off on that. The next thing I'm trying is what was in this thread on reddit - as what they are describing therein is exactly what I've been experiencing.

As an aside, I'm hoping that someone at HE reads the report that you sent in via email, or this or any of the other threads regarding any of the problems pertaining to the tunnel service, and is able to determine a cause and provide a resolution.

Edit: That did not work. I am trying a different set of settings based on this thread on the Ubiquiti board.

Edit 2: I made the following changes and re-enabled my tunnel GIF interface. So far, the follow settings for me seem to have had a positive effect:

• Tunnel MTU: 1480
• Interface MTU: 1480
• Interface MSS: 1420

Tried the values in the 2nd edit and am seeing a positive result here as well. Thank you!

I haven't heard back at all from Hurricane Electric via email unfortunately, even after sending a follow up. I will send another message though stating that the above values produced a result.