Windows Server 2012 R2, Windows Server 2016 and Windows 10 - no Windows Update?

[AndrewButterworth]

For a few days (I think?) Windows Server 2012 R2, Windows Server 2016 and Windows 10 machines are failing to connect to Windows Update over IPv6.  I get a couple of different errors but looking at the logs it appears it is timing out on connecting to the Windows Update services over IPv6.
If I disable IPv6 and force them to use IPv4 Windows Update works fine.  There are a bunch of DNS AAAA records and Aliases that get resolved so I am not sure which hosts they are failing to connect to.  Connections to other IPv6 sites seem to work fine (google, cisco).
My IPv6 tunnel terminates on the Paris server so was wondering if this is anything related to the hardware failure the other week and the temporary terminating of the Paris tunnels in London?

Andy

[AndrewButterworth]

It seems the Windows 2012 R2, Windows 2016 and Windows 10 machines connect to sls.update.microsoft.com (alias to sls.update.microsoft.com.nsatc.net) and Windows 7 connect to fe2.update.microsoft.com (alias to fe2.update.microsoft.com.nsatc.net).  Windows 7 (and server 2008 R2) clients work OK, the others don't.  I can see connections being made and the remote servers responding so I don't think its a routing issue.  It could be a MTU issue along the path (had this before with cisco.com) so I might try lowering the MTU on a client.
There is a mix of OEM & KMS licensed machines - however I don't think this is relevant if it works over IPv4?
Could it be geolocation stuff as my IPv4 address space is UK and my IPv6 is French?

Anyone else seeing this behaviour?

Andy

[cholzhauer]

I don't have this problem on my Windows 10 machine at home, nor have I seen it with people that have native connectivity, so you may be on to something with the MTU.

(I am on a different tunnel server than you)

[kriteknetworks]

I'm on CHI tserv, from my Windows 10 VM I tried

curl -v -6 https://sls.update.microsoft.com

and it just sits there, eventually timing out. Using -4 arg works fine.

[AndrewButterworth]

OK.  So I have a couple of Squid proxy servers that my Windows hosts use via autodiscovery.  These sit on a VLAN where I have lowered the IPv6 MTU to 1280 and Windows Update now works for the Server 2012 R2, 2016 and Windows 10 hosts.

I tried the curl command kriteknetworks posted on a Windows host on a 'normal' VLAN with a 1500 byte MTU and this starts OK but times out:

Code Select Expand

C:\Windows\system32>curl -v -6 https://sls.update.microsoft.com
* Rebuilt URL to: https://sls.update.microsoft.com/
*   Trying 2a01:111:f307:1790::f001:7a5...
* TCP_NODELAY set
* Connected to sls.update.microsoft.com (2a01:111:f307:1790::f001:7a5) port 443
(#0)
* schannel: SSL/TLS connection with sls.update.microsoft.com port 443 (step 1/3)

* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 197 bytes...
* schannel: sent initial handshake data: sent 197 bytes
* schannel: SSL/TLS connection with sls.update.microsoft.com port 443 (step 2/3)

* schannel: failed to receive handshake, need more data
* Operation timed out after 300115 milliseconds with 0 out of 0 bytes received
* Closing connection 0
* schannel: shutting down SSL/TLS connection with sls.update.microsoft.com port
443
* schannel: clear security context handle
curl: (28) Operation timed out after 300115 milliseconds with 0 out of 0 bytes r
eceived

On the ubuntu squid host with the lower IPv6 MTU it fails but doesn't timeout (it had the same behaviour as the Windows host before I lowered the MTU)

Code Select Expand

administrator@ubuntu-squid-1:~$ curl -v -6 https://sls.update.microsoft.com
* Rebuilt URL to: https://sls.update.microsoft.com/
*   Trying 2a01:111:f307:1790::f001:7a5...
* Connected to sls.update.microsoft.com (2a01:111:f307:1790::f001:7a5) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 596 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
administrator@ubuntu-squid-1:~$

It therefore looks like somewhere in the path to the microsoft IPv6 hosts (sls.update.microsoft.com.nsatc.net) there is an MTU issue - a router blocking ICMP packet too big maybe?
I am not quite sure how we get this looked into further as it obviously something outside of my control or influence?

Andy

[AndrewButterworth]

Found a post on the MS technet forums regarding this as well:

https://social.technet.microsoft.com/Forums/windowsserver/en-US/16e7aa06-9b90-48f8-8370-76c2329b93a8/windows-update-ipv6?forum=ws2016

Looks like it is a MS (or their CDN provider) issue.

[tomkep]

This works for me for Warsaw tunnel and curl to https://sls.update.microsoft.com:

Code Select Expand

ip6tables -t mangle -A POSTROUTING -d 2a01:110::/31 -o he-net -p tcp --syn -j TCPMSS --set-mss 1392

I am on ppp-oe, which takes 8 bytes from regular ethernet MTU, so you may try 1400, it just may work for you.

[AndrewButterworth]

I have a FTTP connection with PPPoE (IPv4) and negotiate an MRU of 1500 as it uses mini-Jumbo frames (the routers parent Ethernet interface has a MTU of 1508).  The IPv6 Tunnel has a MTU of 1480 and I have reduced the IPv6 MTU to 1440 on the VLAN SVI interface where the Ubuntu Squid servers are (this obviously gets announced in RA's so the hosts know the IPv6 MTU).
I have just been playing around with MTUPATH.exe to try and work out the maximum IPv6 MTU along the path.  I can set the VLAN SVI to 1480 and this works so I guess the Ubuntu hosts are getting the ICMP packet-to-big's locally and this is working around the issue for me. Somewhere along the path the ICMP packet-to-big's are either not being sent or blocked.

[AndrewButterworth]

Looks like this is now working.  Not sure who did what?

[tomkep]

It is NOT working for me.

[kriteknetworks]

Working for me now too, didn't change anything....

curl -k -6 https://sls.update.microsoft.com

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>403 - Forbidden: Access is denied.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
<div class="content-container"><fieldset>
  <h2>403 - Forbidden: Access is denied.</h2>
  <h3>You do not have permission to view this directory or page using the credentials that you supplied.</h3>
</fieldset></div>
</div>
</body>
</html>

[mkyplk]

It still does not work for me. Has someone solved this problem?