Significant packet loss in he HE ipv6 net?

[hawk7000]

I'm seeing high packet loss in the path NYC <-> EU now and it has been coming and going for a while (a week?).

As seen from the NYC side:

$ mtr --report-wide --report ping.sunet.se
HOST: a.qw.se                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. hawk7000-2.tunnel.tserv4.nyc4.ipv6.he.net  0.0%    10    5.6   5.6   4.9   6.1   0.4
  2. gige-g3-8.core1.nyc4.he.net                0.0%    10    1.8   4.0   1.7  12.1   4.1
  3. 10gigabitethernet3-3.core1.lon1.he.net     0.0%    10   79.5  73.7  69.9  80.4   4.7
  4. 10gigabitethernet1-1.core1.ams1.he.net    50.0%    10   85.0  81.7  80.8  85.0   1.8
  5. 10gigabitethernet1-1.core1.fra1.he.net    40.0%    10   88.2  91.2  87.6 104.7   6.7
  6. 10gigabitethernet1-1.core1.sto1.he.net    50.0%    10  112.4 112.2 111.9 112.4   0.2
  7. 2001:7f8:d:fb::19                         30.0%    10  112.6 112.7 112.2 113.1   0.3
  8. ping.sunet.se                             30.0%    10  112.3 112.5 112.0 112.6   0.2
$

As seen from a host on the other side:

$  mtr --report --report-wide a.qw.se
HOST: h7k                                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- hawk7000-1.tunnel.tserv24.sto1.ipv6.he..7.2.0.0.0.7.4.0.1.0.0.2.ip6.arpa  0.0%    10    8.5   9.3   8.5  10.4   0.6
  2.|-- gige-g2-20.core1.sto1.he..0.0.0.e.1.1.0.0.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa  0.0%    10    7.2   9.1   7.1  15.3   2.6
  3.|-- 10gigabitethernet3-3.core1.fra1.he..1.0.0.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa  0.0%    10   32.9  37.6  32.9  47.7   4.9
  4.|-- 10gigabitethernet1-4.core1.ams1.he..0.0.0.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa  0.0%    10   42.6  40.2  38.8  42.6   1.1
  5.|-- 10gigabitethernet1-4.core1.lon1.he..0.0.0.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa  0.0%    10   47.1  50.5  46.1  56.4   3.4
  6.|-- 10gigabitethernet4-4.core1.nyc4.he..1.0.0.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa 10.0%    10  117.0 121.1 117.0 130.3   4.8
  7.|-- gige-gbge0.tserv4.nyc4.ipv6.he..d.5.0.0.0.0.0.0.0.7.4.0.1.0.0.2.ip6.arpa 10.0%    10  121.9 123.1 121.5 126.6   1.4
  8.|-- a.qw..0.0.0.0.0.0.0.0.0.0.0.0.0.a.d.5.0.7.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa 20.0%    10  119.5 120.1 119.5 120.9   0.6
$

So the problem shows a bit assymetrically, but there's a lot of packet loss no matter how you look at it.

Anything going on?

[mleber]

An optic on one of the 10G circuits between New York and London failed this morning, the wavelength involved was moved to a different port and brought back up.

Everything should be fully operational now.

We have many US, translatlantic, and European circuits pending delivery that will nicely improve the network in the next few weeks.

[jimb]

Heh gotta love HE.  Response from the CEO!

[hawk7000]

An optic on one of the 10G circuits between New York and London failed this morning, the wavelength involved was moved to a different port and brought back up.

Everything should be fully operational now.

We have many US, translatlantic, and European circuits pending delivery that will nicely improve the network in the next few weeks.

Ok, thanks for clarifying what's up!