Seattle IPv4 PoP Down

[curriegrad2004]

It's been down as of 11:30PM PST.

Seems like it's stopped at the core router over at Seattle:

Code Select Expand

Tracing route to tserv1.sea1.he.net [216.218.226.238]
over a maximum of 30 hops:

  1     3 ms     2 ms     2 ms  van1.router.smb.curriegrad2004.ca [10.0.7.65]
  2    37 ms    65 ms    81 ms  10.31.192.1
  3    31 ms    31 ms    33 ms  75.154.216.254
  4    41 ms    36 ms    42 ms  gige-g2-10.core1.sea1.he.net [66.160.143.117]
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8  ^C

[bytesize]

+1

[intercal]

Same here.  I can't ping or ping6 the tunnel server - tried from a few places with different ISPs
The status page claims it's up, but looks like it's not been updated.

From Seattle:

Code Select Expand


traceroute to 216.218.226.238 (216.218.226.238), 64 hops max, 40 byte packets
1  vl160-d2.acc.sea2.hopone.net (192.94.73.3)  0.675 ms  0.522 ms  0.372 ms
2  ge4-2.core2.sea2.hopone.net (209.160.60.221)  2.400 ms  1.169 ms  0.215 ms
3  xe1-0.core1.sea1.hopone.net (66.36.224.18)  1.366 ms  1.278 ms  2.749 ms
4  10gigabitethernet1-3.core1.sea1.he.net (206.81.80.40)  1.224 ms  3.981 ms  1.216 ms
5  * * *
6  * * *
^C

(can't traceroute6 as that would require the tunnel to be up!)

And from the UK:

Code Select Expand


$ traceroute tserv1.sea1.he.net
traceroute to tserv1.sea1.he.net (216.218.226.238), 30 hops max, 60 byte packets
1  srv.i.w42.org (81.187.48.225)  0.413 ms  0.336 ms  0.257 ms
2  c.gormless.thn.aa.net.uk (90.155.53.53)  21.345 ms  22.192 ms  22.737 ms
3  a.aimless.thn.aa.net.uk (90.155.53.41)  23.880 ms  24.434 ms  25.335 ms
4  40gigabitethernet1-1.core1.lon1.he.net (195.66.224.21)  26.768 ms  27.405 ms  32.276 ms
5  10gigabitethernet10-4.core1.nyc4.he.net (72.52.92.241)  98.286 ms  99.096 ms  99.447 ms
6  100gigabitethernet13-2.core1.chi1.he.net (184.105.223.161)  123.598 ms  106.656 ms  108.238 ms
7  10gigabitethernet4-1.core1.den1.he.net (72.52.92.234)  132.336 ms 10gigabitethernet3-2.core1.den1.he.net (184.105.213.86)  131.045 ms  131.933 ms
8  10gigabitethernet3-4.core1.sea1.he.net (184.105.213.41)  177.216 ms  177.641 ms  177.565 ms
9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  *^C

traceroute to tserv1.sea1.he.net (2001:470:0:9b::2), 30 hops max, 80 byte packets
1  srv.i.w42.org (2001:8b0:1a7:4307::1)  0.637 ms  0.599 ms  0.535 ms
2  c.gormless.thn.aa.net.uk (2001:8b0:0:53::53)  21.672 ms  22.015 ms  23.252 ms
3  2001:7f8:4::50e8:1 (2001:7f8:4::50e8:1)  24.147 ms  25.088 ms  25.931 ms
4  40gigabitethernet1-1.core1.lon1.he.net (2001:7f8:4::1b1b:1)  27.077 ms  28.561 ms  28.463 ms
5  10gigabitethernet10-4.core1.nyc4.he.net (2001:470:0:128::1)  98.447 ms  99.013 ms  99.352 ms
6  100gigabitethernet13-2.core1.chi1.he.net (2001:470:0:298::1)  119.566 ms  106.867 ms  118.597 ms
7  10gigabitethernet3-2.core1.den1.he.net (2001:470:0:1af::2)  132.781 ms 10gigabitethernet4-1.core1.den1.he.net (2001:470:0:293::2)  129.981 ms  131.182 ms
8  10gigabitethernet5-1.core1.den1.he.net (2001:470:0:240::1)  142.920 ms 10gigabitethernet3-4.core1.sea1.he.net (2001:470:0:18f::1)  165.272 ms  166.133 ms
9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  *^C


[Pyker]

+1.

I can't ping it either.

Here's the traceroute:

Code Select Expand

traceroute to 216.218.226.238 (216.218.226.238), 30 hops max, 60 byte packets
1  rtr2.vl24.lw.netriver.net (69.46.39.2)  1.104 ms  1.179 ms  1.209 ms
2  10gigabitethernet1-3.core1.sea1.he.net (206.81.80.40)  2.856 ms  2.853 ms  2.809 ms
3  * * *
(...)

[cholzhauer]

Did anyone email ipv6@he.net ?

[Pyker]

Did anyone email ipv6@he.net ?

I haven't because I thought the OP had.
Anyway, it shows as "down" on the status page, so I think they're already aware.

EDIT: Just in case, I just sent one.

[jjulian]

Support response:

I don't have an ETA on that.  Looks like a possible hardware fault.  If
so, there likely wouldn't be a replacement until next week, and in the
short term (today), we'd rehome tunnels to the next closest available
tunnel server with space.  At the moment, that would appear to be kind
of a toss-up between Fremont or Denver.

[jjulian]

Any chance the re-homing will be done in the next hour? Are you just going to move the ipv4, or will I need to change my gateways?

We'd just assign the appropriate IPs and tunnel configs to the other tunnel server.  You'd just start to see the tunnel work again.  As to timing, still trying a few more things.

[kcochran]

Seattle tunnels have been rehomed to Denver until the Seattle tunnel server is back in operation.  You shouldn't have to do anything on your side to adjust for this change.

[arader]

newb question: would this be why I can't ping my machines from external addresses? I'm in seattle and I'm able to access IPv6 from my network, but if I use the IPv6 portscan (http://tunnelbroker.net/portscan.php) it's unable to ping anything in my prefixes.

Note that this was working fine yesterday, I was able to ping and ssh using v6. today? nothing.

[kcochran]

Tunnels are back to Seattle now.

[kasperd]

would this be why I can't ping my machines from external addresses? I'm in seattle and I'm able to access IPv6 from my network, but if I use the IPv6 portscan (http://tunnelbroker.net/portscan.php) it's unable to ping anything in my prefixes.

When a tunnel server is down, that's going to affect traffic regardless of which side it is established from. Depending on the nature of the outage, it could possibly affect only packets in one direction. But any connection you use is going to depend on packets in both directions.

More likely explanations for being able to open connections from your network and out but not connections coming in, is that you have a firewall configured to treat traffic that way. It may also be that your tunnel is going through a NAT device and thus cannot get incoming traffic unless there has been some outgoing traffic recently. I can't come up with an explanation for why that would correlate with an outage on the tunnel server. It would not surprise me if a tunnel server outage combined with a NAT device could have triggered the problem you see.

Could you try a connection to reflector.easyv6.net? It is a special IPv6 address that will bounce all TCP traffic back to you. So if you have an open TCP port and try to connect to that port number on reflector.easyv6.net, you should get back to your own machine.