Lots of Akamai IPv6 ICMP on IPv6 Tunnel

[takio]

Hi,

the last 2-3 days i see constant traffic on my he ipv6 tunnel interface (Frankfurt). Its ipv6 traffic from over 1000 different IPv6 IP's, all of them owned by Akamai.
My first tought was thats some sort of P2P Traffic from Akamai download accelerators. So i checked my systems, the accelerator is not installed, and there is no IPv6 traffic on my LAN.

The IPv6 traffic points to my HE.net Tunnelendpoint, i can't see any traffic on my /48 subnet i use for my LAN.

After 3 minutes of traffic scan, i got 638 hosts sending icmp packages to my tunnelendpoint.

Thats a lot, but very low traffic for some sort of DOS, and no, there is no dns traffic (amp attack).

Any ideas what this traffic from Akamai is?


Here is a screenshot of my traffic sniff.

[kasperd]

Akamai might be the largest CDN with IPv6 support, so seeing lots of traffic exchanged with Akamai IP addresses is really not unexpected.

You might not be able to correlate the traffic you see from Akamai with packets send from your own network. I still think the most likely explanation is, that it is somehow correlated. It is entirely possible Akamai is doing something unobvious or even incorrect with the traffic, so don't feel bad about not spotting any correlation. I still think with sufficient knowledge about the packets, we can figure out, which packets you send triggers those incoming packets.

My first question is, what are those ICMPv6 packets, which you receive from Akamai? Are they echo requests? Echo replies? Error messages? Something else?

[takio]

Hi,

i removed my local lan (to be sure its not produces from inside my lan) and made a pcap on the tunnel interface. I loaded the pcap in Wireshark and found that all these hosts send me an echo ping request, that my router respond with an echo reply.  :-\

[kasperd]

I loaded the pcap in Wireshark and found that all these hosts send me an echo ping request, that my router respond with an echo reply.

Could you attach the pcap file to this thread, then I can take a look at the echo requests and remote IP addresses. That way I might be able to get a slightly better idea, what is producing the requests.

There could be good reasons for sending such echo requests. For example it might help measure connectivity to more reliably deliver content. Such probing could be looking both at packet loss and roundtrip times. It might also help identify the location of hosts in order to produce better geolocation.

But regardless of the purpose, it definitely shouldn't be the majority of traffic. If you get all these echo requests and reply to them, and don't see any other IPv6 traffic, then something weird is going on.

A few pieces of additional information, which might be useful is the following:
- Are you able to ping those addresses? (Pick a handful of different addresses from the trace at random and try to ping each)
- Do you have any AAAA record pointing to the destination IP of those echo requests?
- Has this been going on for the entire time you have had the tunnel?

[takio]

I loaded the pcap in Wireshark and found that all these hosts send me an echo ping request, that my router respond with an echo reply.

A few pieces of additional information, which might be useful is the following:
- Are you able to ping those addresses? (Pick a handful of different addresses from the trace at random and try to ping each)
- Do you have any AAAA record pointing to the destination IP of those echo requests?
- Has this been going on for the entire time you have had the tunnel?

1. Yes i can ping these adressess
2. Not by myself. Its has the default AAAA and reverse every tunnelendpoint has.
3. No, this tunnel is several years old. I sniff this interfaces regulary to find network issues or testing stuff. This ICMP traffic startet about a week ago.

I have attached a capture i made on my Microtik Router.

Btw. i checked the filter statistics for ICMPv6, i got around 190mb / 2.718.230 Packets since reboot (2 Weeks Ago), i would say thats alot ;)

[kasperd]

The echo requests look pretty ordinary. They might very well have been produced by a standard ping command. The IP addresses these packets originate from does not appear to be webservers (I only checked a couple of them though). So all I have learned from looking at the trace is, that there are not many hints to be found from looking at the trace.

I did find that in the trace there are sequences of multiple echo requests with identical ID from the same source IP with apparently random sequence numbers. That is also a behaviour you could expect to see from a standard ping command. I looked at one such sequence with a 30 second interval (except that it jumped to 35 seconds once). I looked at another such sequence with a 15 second interval. Except from the occasional jump in interval, this is also something, which could have been produced by a standard ping command.

I also checked a couple of source addresses to see, that they were really from Akamai. Both addresses appear to be associated with Akamai, though for one of them it was not obvious. If anybody likes puzzles, look at 2a02:d28:101:908::4e98:2be4 and find the clue leading from that IP to Akamai.

My best guess is that somebody at Akamai created a tunnel in order to measure connectivity, and then mistyped the IP address of the tunnel they had created, and typed yours instead. You could try to send the packet trace to abuse@akamai.com and ask if they mistyped an IP address. (I got that IP from whois on one of the IP addresses from your trace).

[takio]

Hi,

I already contacted them 2 days ago at abuse@akamai.com and describing them the issue including the capture.

Behind 2a02:d28:101:908::4e98:2be4 is a transit carrier, a good place to place cloud servers, maybe they use different ip space on this one.

Thanks for your help :)

[takio]

I'am still getting all this icmp echo requests from Akamai, no response at abuse@akamai.com.

Thanks to RouterOS, its easy to build an addresslist and use it to filter/block traffic. I now block icmp6 for around 1250 Akamai hosts... lol.

[kasperd]

I now block icmp6 for around 1250 Akamai hosts... lol.

That's not very creative. I would have done the following:

• Sort the list of IPv6 addresses
• Split the list into the 625 first addresses on the sorted list and the 625 last addresses on the sorted list
• For requests from one of the two lists, reply to all requests as fast as you can.
• For requests from the other list, reply to only 95% of the received packets, and delay each reply by 50ms.
• Every 13 hours swap the two lists.

If the requests are really due to somebody at Akamai mistyping an IP address, then that is sure to get somebody to scratch their head trying to explain the strange graphs changing every 13 hours.

It occurs to me, that there could be another explanation. If Akamai is running a service similar to this, then the packets could be explained by a user of the service mistyping an IP address rather than somebody at Akamai. I do not know, if Akamai has any such service.

[takio]

After dropping Akamai's icmpv6 requests for a few days, it has stopped :)

[kasperd]

After dropping Akamai's icmpv6 requests for a few days, it has stopped :)

And we might never know, if it would have stopped anyway. ;D