tserv1.ams1.he.net packet loss

[ictd01]

Hello,

I did notice 25% packet loss at the moment. I am located in The Netherlands and connected to tunnel server 216.66.84.46 tserv1.ams1.he.net.

Am I the only one? Or are there more having connectivity issues?

Thank you.

[ping]

You are not alone, i am also experiencing issues with my tunnel (also via 216.66.84.46).

Altho im not 100% sure its related to the tunnelserver, or a just a local issue with my hardware, but after sniffing a lot of packets over the last couple hours, i noticed that sometimes, the packets received on the tunnel interface, has a few bits flipped.

For example 0x11 would become 0x10, or 0x74 becomes 0xf4 etc.
Bitrot-over-ipv6? :P

Often this hits dns replies, with the encapsuled packet having a Next-Header value of 0x10, instead of 0x11 (IP protocol CHAOS, instead of UDP).

I dont know when this started, but i first noticed it about ~7-8 hours ago (around 20.30 CET)..

It more or less breaks ipv6 connectivity completely, as most packets fail checksums, has bogus headers, wrong ip adresses or similar, as a result of the flipped bits.

..Again, it could just be a local issue with my hardware, but it smells like its tunnelserver-related..

[ictd01]

I run pfSense and I did notice this quite a lot:

Aug 21 14:30:32   dpinger   83013   IPV6WAN1_TUNNELV6 2001:470:1f14:71::1: duplicate echo reply received
Aug 21 14:21:53   dpinger   83013   IPV6WAN1_TUNNELV6 2001:470:1f14:71::1: duplicate echo reply received
Aug 21 13:29:50   dpinger   83013   IPV6WAN1_TUNNELV6 2001:470:1f14:71::1: duplicate echo reply received

That's suspicious.

[ictd01]

I have sent an email to ipv6@he.net. Hopefully we'll soon find out what's causing it.

[ping]

I run pfSense and I did notice this quite a lot:

Aug 21 14:30:32   dpinger   83013   IPV6WAN1_TUNNELV6 2001:470:1f14:71::1: duplicate echo reply received
Aug 21 14:21:53   dpinger   83013   IPV6WAN1_TUNNELV6 2001:470:1f14:71::1: duplicate echo reply received
Aug 21 13:29:50   dpinger   83013   IPV6WAN1_TUNNELV6 2001:470:1f14:71::1: duplicate echo reply received

That's suspicious.

My tunnel is configured on BSD too, but with ipfw instead of pf..it too complains quite a lot in the logs..not about dups, but about unknown protocols (because 0x11 is flipped into 0x10 in the packet):

Aug 21 16:00:38 <kern.crit> kernel: IPFW2: IPV6 - Unknown Extension Header(16), ext_hd=0
Aug 21 16:01:05 <kern.crit> kernel: IPFW2: IPV6 - Unknown Extension Header(16), ext_hd=0
Aug 21 16:02:11 <kern.crit> kernel: IPFW2: IPV6 - Unknown Extension Header(16), ext_hd=0
Aug 21 16:13:27 <kern.crit> last message repeated 12 times
Aug 21 16:24:01 <kern.crit> last message repeated 9 times

Once in a while, it hits the source ip in the packet instead, so for example 2a01:4f8:120:: becomes 2901:4f8:120::
I have never experienced a bug quite like this, its kinda interesting..and annoying :P

[ictd01]

I did notice that there is no loss anymore since 22-08-2022 — after 11:08:35. It looks like the problem has been resolved :-)

[ping]

I did notice that there is no loss anymore since 22-08-2022 — after 11:08:35. It looks like the problem has been resolved :-)

Wooo! :D
I didnt even notice it had stopped, i had to fall back to ipv4 on most of my gear, because it was completely useless otherwise.
Whatever it was, seems so have stopped flooding my log too, roughly 2 hours later than you (times in CET)

Aug 22 13:03:55 <kern.crit> kernel: IPFW2: IPV6 - Unknown Extension Header(16), ext_hd=0
Aug 22 13:06:36 <kern.crit> last message repeated 28 times

Im super curious about what/how/where it went wrong..
Did you get any reply with info about what had happened? ???