Frequent intermittent outages on tserv15.lax1

[rcarroll]

Has anyone noticed brief outages on tserv15? I keep getting kicked off gtalk and I thought it was Google at first. But I seem to be losing IPv6 connectivity upstream. So far I saw outages today at 8:09 am PDT, 8:33 am PDT, and 10:08 am PDT.

I verified the tunnel was up, and I could ping the remote endpoint just fine. This is, well, Google...

Code Select Expand

bob@zim ~ $ ping6 ipv6.google.com
PING6(56=40+8+8 bytes) 2001:470:f4ce:1:129a:ddff:fe57:a504 --> 2607:f8b0:4007:801::1014
Request timeout for icmp_seq=0
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=1 hlim=58 time=374.441 ms
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=2 hlim=58 time=418.987 ms
Request timeout for icmp_seq=3
Request timeout for icmp_seq=4
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=5 hlim=58 time=410.072 ms
Request timeout for icmp_seq=6
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=7 hlim=58 time=418.140 ms
Request timeout for icmp_seq=8
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=9 hlim=58 time=448.944 ms
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=10 hlim=58 time=427.722 ms
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=11 hlim=58 time=410.763 ms
Request timeout for icmp_seq=12
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=13 hlim=58 time=449.381 ms
Request timeout for icmp_seq=14
16 bytes from 2607:f8b0:4007:801::1014, icmp_seq=15 hlim=58 time=462.309 ms

And this guy has a SixXS tunnel...

Code Select Expand

bob@zim ~ $ ping6 tthanna.enimihil.net
PING6(56=40+8+8 bytes) 2001:470:f4ce:1:129a:ddff:fe57:a504 --> 2001:4978:22c::92fb:a6ff:fe33:2ebb
Request timeout for icmp_seq=0
16 bytes from 2001:4978:22c::92fb:a6ff:fe33:2ebb, icmp_seq=1 hlim=244 time=520.046 ms
Request timeout for icmp_seq=2
Request timeout for icmp_seq=3
Request timeout for icmp_seq=4
Request timeout for icmp_seq=5
16 bytes from 2001:4978:22c::92fb:a6ff:fe33:2ebb, icmp_seq=6 hlim=244 time=541.546 ms
Request timeout for icmp_seq=7
Request timeout for icmp_seq=8

Unfortunately I didn't think to do a traceroute, but I'll add that here next time it happens.

[snarked]

I have not noticed any timeouts or such, but I also wasn't on during those times from home, and I nothing unusual in my colocated server's logs (separate tunnel - same tunnel server).  As of 11:15 a.m., all appears normal.

[willphase]

mine just went down... still down. oh no, it's back... but it was down for at least 2-3 mins

gige-g4-8.core1.lon1.he.net [2001:470:0:67::1]

Will

[mhisani]

Confirm same problem randomly London

[cholzhauer]

Been seeing the same problem on Chicago too, although the problem only seems to happen in the afternoon.

[mtindle]

I'm not seeing any problems from here.  We'd need v6 traceroutes (or better, mtrs) to the endpoint and would also like to see v4 traceroutes / mtrs to your tunnel endpoint as well.  Please feel free to send them into ipv6@he.net if you continue to see issues and we'll look into it further. 

[kasperd]

We'd need v6 traceroutes

What kind of monitoring does HE currently do on the tunnel servers?

What I would do if I was responsible for such a service would include the following:
1. Setup a machine with IPv4 connectivity (IPv6 connectivity is optional).
2. On that machine run 31 virtual machines with only IPv4 connectivity.
3. From each virtual machine create a tunnel to each tunnel server.
4. Replicate this setup in two or three locations.
5. Run periodic IPv6 traceroutes between pairs of virtual machines.
6. Alert if multiple virtual machines using the same tunnel server are experiencing connectivity problems.