he.net DNS outage 03 Jan 2011

[packetmail]

Hello, first I fully understand the ramifications of an "Open Beta" on HE's DNS open to those participants of the 6in4 tunnel.  This isn't a complaint or disdain but I'd like to understand more about what went wrong and why as I've moved DNS to he.net as out-of-bailiwick for a pure IPv6 presence.  The outage affected IPv4 (obviously).  Sure I could plant quad-A's in an IPv4 only nameserver but the goal was early adoption/understanding and a pure IPv4/IPv6 Internet presence.

From what I understand, around 8:00 EST https://dns.he.net/ reported "Down for maintenance".
DNS was sporadic some time after, impacting tunnelbroker.net, he.net, and pretty much anything sitting on ns[1-5].he.net.
Some time later, perhaps around 10:00 EST https://dns.he.net/ reported (paraphrased) "Nameservers are setting heavy traffic and an engineer is investigating"
Service was restored around 17:00 EST with https://dns.he.net/ being updated to reflect the normal presence about ~1 HR after resolution.

(note error margins on timescale are +- 1 HR)

Was the heavy traffic related to the previous maintenance?  How often do these issues persist or manifest; I would imagine not often at all since the scope of impact was quite large affecting he.net as well as tunnelbroker.net?

Any information you're willing to share?  Again, you're the supplier of my 6in4 and nameservers, I have no SLA, I pay nothing, and I eat your bandwidth.  I have no sense of entitlement but I need to assess stability versus IPv6 presence and appreciate any replies or additional information.  Even a "things went crazy, last time this happened was 9 months ago" is valuable.

[broquea]

The DDOS of our nameservers has been mitigated.  The nature of the attack was such that there was not a single source which made blocking the attack very difficult and reporting it even more so.  There were hundreds of thousands of sources.  Moving forward, we are re-engineering our nameserver infrastructure to better cope with attacks such as this.

Over the coming months we will be deploying an 25 additional nameservers that will be anycasted globally thus providing us additional redundancy and resilience against attacks.  6 servers were deployed last night and will be brought live later today.

[packetmail]

broquea, thank you for taking the time to respond and provide detailed information.  I was not aware it was a DDoS (with a connotation towards being malicious).  This changes things dramatically, I suspected perhaps it was a errant client configuration or other resolver-saturating behavior.  A DDoS changes things dramatically and makes the outage more palatable/understandable.

I appreciate the prompt response and level of detail.  Again, I appreciate the free services provided by HE, specifically the certification and educational resources provided, and the overall technical proficiency exhibited by the HE staff.

Cheers

[cholzhauer]

Yeah, same here...I had no idea that HE was under a DDoS attack (I don't use their DNS servers though)

It is interesting to read about things though

[Ninho]

Happy new year to you all !

And, awe...  ??? ! An attack from "hundred of thousands" of sources ? That's sure looking huge. Or were those (spoofed) source IPs coming from a lesser number of actual sources ?

I sure would like to read more details of a technical nature that you could provide without breaking confidentiality. Was is aimed at only Hurricane Electric, or a general DDOS on IPv6 networks ? Who could be willing to "attack" you, or take IPv6 down (I didn't think IPv6 services were important enough yet to not be under the bad guys' radars. Unless they, too, are in training...)?

Was that specifically attacking DNS ? Using known patterns ?

I'm sure any details you can and care to share would be of interest not just to myself, many of the fine people here...

--
Ninho

[allen4names]

I suspect that zombies were used.  While I hope this does not happen again it is best to be prepared.

[jimb]

FWIW:  http://www.arbornetworks.com/