1443595 - github binary downloads are broken in occ due to the tls upgrade

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Comment 1

[Pete Moore [:pmoore][:pete]]

It looks like github requires TLS 1.2 and the Net.ServicePointManager is configured to use TLS 1.0.

Switching to TLS 1.2 should do the trick:


Windows PowerShell
Copyright (C) 2014 Microsoft Corporation. All rights reserved.

Choose arbitrary file to download from github:

> PS C:\Users\Administrator> $client = New-Object system.net.WebClient
> PS C:\Users\Administrator> $client.DownloadFile("https://github.com/taskcluster/generic-worker/releases/download/v10.6.0/generic-worker-windows-amd64.exe", "C:\generic-worker.exe")
> Exception calling "DownloadFile" with "2" argument(s): "The request was aborted: Could not create SSL/TLS secure channel."
> At line:1 char:1
> + $client.DownloadFile("https://github.com/taskcluster/generic-worker/releases/dow ...
> + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
>     + FullyQualifiedErrorId : WebException

Check which version of TLS we are using:

> PS C:\Users\Administrator> [Net.ServicePointManager]::SecurityProtocol
> Ssl3, Tls

This presumably means TLS 1.0. TLS 1.2 is the newest (see https://tlsversions.com/) - so let's upgrade to this.

> C:\Users\Administrator> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Ssl3,[Net.SecurityProtocolType]::Tls12

Check the update worked:

> PS C:\Users\Administrator> [Net.ServicePointManager]::SecurityProtocol
> Ssl3, Tls12

Now check again if we can download the resource that we couldn't download before:

> PS C:\Users\Administrator> $client.DownloadFile("https://github.com/taskcluster/generic-worker/releases/download/v10.6.0/generic-worker-windows-amd64.exe", "C:\generic-worker.exe")
> PS C:\Users\Administrator> ls C:\generic-worker.exe
> 
> 
>     Directory: C:\
> 
> 
> Mode                LastWriteTime     Length Name
> ----                -------------     ------ ----
> -a---          3/7/2018   2:33 PM   14706688 generic-worker.exe
> 
> 
> PS C:\Users\Administrator>

Success!


I'll make a patch to rundsc.ps1 to use TLS 1.2 before doing anything else.

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for OpenCloudConfig

This should fix the TLS issues.

Tested on a standard AWS Windows Server 2012 R2 image ("Windows_Server-2012-R2_RTM-English-64Bit-Base-2018.02.23 (ami-014a7d64)").

The test instance I used was: https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#Instances:search=i-0a7a81074ef97bec1;sort=instanceId

Comment 3

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #1)

> This presumably means TLS 1.0. TLS 1.2 is the newest (see
> https://tlsversions.com/) - so let's upgrade to this.

Indeed - "Tls" does refer to TLS 1.0, as suspected:

https://msdn.microsoft.com/en-us/library/system.net.securityprotocoltype%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396

Note - https://tlsversions.com/ suggests not using SSL3 at all, so I the patch just allows TLS 1.2 and not allow SSL3. In other words, the change is:

> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

rather than:

> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Ssl3,[Net.SecurityProtocolType]::Tls12

in order that SSL3 is not permitted.

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/generic-worker

https://github.com/taskcluster/generic-worker/commit/b3b357ab1d96f1f9a9f7ece0565de9e57821c249
Bug 1443595 - use TLS 1.2 when downloading from HTTPS

Comment 5

[Pete Moore [:pmoore][:pete]]

(In reply to [github robot] from comment #4)
> Commit pushed to master at https://github.com/taskcluster/generic-worker
> 
> https://github.com/taskcluster/generic-worker/commit/
> b3b357ab1d96f1f9a9f7ece0565de9e57821c249
> Bug 1443595 - use TLS 1.2 when downloading from HTTPS

Note - this change is to support *non-gecko* worker types that would potentially hit the same issue that OCC had (e.g. nss-win2012r2, win2012r2 worker types).

Comment 6

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8956830 [details] [diff] [review]
Github Pull Request for OpenCloudConfig

Review of attachment 8956830 [details] [diff] [review]:
-----------------------------------------------------------------

i'm happy with this as long as we've tested it on all three platforms (7, 10, 2012)

Comment 7

[Pete Moore [:pmoore][:pete]]

(In reply to Rob Thijssen (:grenade UTC+2) from comment #6)
> i'm happy with this as long as we've tested it on all three platforms (7,
> 10, 2012)

I'm not sure how to test it in rundsc.ps1 without that change being propagated to all worker types, but I have tested from a powershell console on all our supported Windows OS versions. I'll attach screenshots.

Comment 8

[Pete Moore [:pmoore][:pete]]

Attachment: bug1443595-win7.png

Windows 7 Powershell screenshot.

Comment 9

[Pete Moore [:pmoore][:pete]]

Attachment: bug1443595-win10.png

Windows 10 Powershell screenshot.

Comment 10

[Pete Moore [:pmoore][:pete]]

Attachment: bug1443595-win2012r2.png

Windows Server 2012 R2 Powershell screenshot.

Comment 11

[Pete Moore [:pmoore][:pete]]

Hey Rob,

Do you think this is safe to land, based on those screenshots, or is some other form of testing required? I'm not sure how to test the full rundsc.ps1 change, without committing it, which I think would then get automatically deployed to all production workers, is that right?

Let me know if you think we should risk committing it, or if there is something else we should do first.

Thanks,
Pete

Comment 12

[Rob Thijssen [:grenade (EET/UTC+0300)]]

just leaving a note to say we discussed on irc last night and are monitoring for fallout or success of the change this morning.

Comment 13

[Pete Moore [:pmoore][:pete]]

Due to bug 1444168 I haven't rolled out today.

Comment 14

[Pete Moore [:pmoore][:pete]]

I rolled out this morning, but it didn't help.

The change doesn't persist across reboots.

Looking at https://stackoverflow.com/questions/28286086/default-securityprotocol-in-net-4-5 we might have some options.

In general I think we could have a problem with .NET 4.0 apps - in particular if DSC runs as a .NET 4.0 app, I'm not sure we'll be able to avoid the issue, because as far as I can see, support for TLS 1.2 was only added with .NET 4.5. Maybe we will need to upgrade the version of Powershell on Windows 10 to use TLS 1.2 from DSC?

It is certainly worth iterating through the suggestions in the stackoverflow link, to see if they get us anywhere.

With my most recent deployments, these are the steps we are currently failing on (may not be exhaustive):

gecko-t-win10-64-beta   [Script]ChecksumFileDownload_GenericWorkerDownload
gecko-t-win10-64-beta   [Script]ChecksumFileDownload_MozFakeCA_2017_10_13_cer
gecko-t-win10-64-beta   [Script]ChecksumFileDownload_MozFakeCA_cer
gecko-t-win10-64-beta   [Script]ChecksumFileDownload_MozRoot_cer
gecko-t-win10-64-beta   [Script]ChecksumFileDownload_maintenanceservice
gecko-t-win10-64-beta   [Script]ChecksumFileDownload_maintenanceservice_installer
gecko-t-win10-64-beta   [Script]CommandRun_GenericWorkerInstall
gecko-t-win10-64-beta   [Script]CommandRun_maintenanceservice_install
gecko-t-win10-64-beta   [Script]FileDownload_LiveLogDownload
gecko-t-win10-64-cu     [Script]ChecksumFileDownload_GenericWorkerDownload
gecko-t-win10-64-cu     [Script]ChecksumFileDownload_MozFakeCA_2017_10_13_cer
gecko-t-win10-64-cu     [Script]ChecksumFileDownload_MozFakeCA_cer
gecko-t-win10-64-cu     [Script]ChecksumFileDownload_MozRoot_cer
gecko-t-win10-64-cu     [Script]ChecksumFileDownload_maintenanceservice
gecko-t-win10-64-cu     [Script]ChecksumFileDownload_maintenanceservice_installer
gecko-t-win10-64-cu     [Script]CommandRun_GenericWorkerInstall
gecko-t-win10-64-cu     [Script]CommandRun_maintenanceservice_install
gecko-t-win10-64-cu     [Script]FileDownload_LiveLogDownload
gecko-t-win10-64-gpu-b  [Script]ChecksumFileDownload_GenericWorkerDownload
gecko-t-win10-64-gpu-b  [Script]ChecksumFileDownload_MozFakeCA_2017_10_13_cer
gecko-t-win10-64-gpu-b  [Script]ChecksumFileDownload_MozFakeCA_cer
gecko-t-win10-64-gpu-b  [Script]ChecksumFileDownload_MozRoot_cer
gecko-t-win10-64-gpu-b  [Script]ChecksumFileDownload_maintenanceservice
gecko-t-win10-64-gpu-b  [Script]ChecksumFileDownload_maintenanceservice_installer
gecko-t-win10-64-gpu-b  [Script]CommandRun_GenericWorkerInstall
gecko-t-win10-64-gpu-b  [Script]CommandRun_maintenanceservice_install
gecko-t-win10-64-gpu-b  [Script]FileDownload_LiveLogDownload
gecko-t-win7-32-beta    [Script]ChecksumFileDownload_GenericWorkerDownload
gecko-t-win7-32-beta    [Script]ChecksumFileDownload_LiveLogDownload
gecko-t-win7-32-beta    [Script]ChecksumFileDownload_MozFakeCA_2017_10_13_cer
gecko-t-win7-32-beta    [Script]ChecksumFileDownload_MozFakeCA_cer
gecko-t-win7-32-beta    [Script]ChecksumFileDownload_MozRoot_cer
gecko-t-win7-32-beta    [Script]ChecksumFileDownload_maintenanceservice
gecko-t-win7-32-beta    [Script]ChecksumFileDownload_maintenanceservice_installer
gecko-t-win7-32-beta    [Script]CommandRun_GenericWorkerInstall
gecko-t-win7-32-beta    [Script]FirefoxBuildSecrets
gecko-t-win7-32-beta    [xArchive]ZipInstall_ProcessMonitor
gecko-t-win7-32-cu      [File]BuildsFolder
gecko-t-win7-32-cu      [File]ChecksumFileCopy_MercurialCerts
gecko-t-win7-32-cu      [File]ChecksumFileCopy_MercurialConfig
gecko-t-win7-32-cu      [File]ChecksumFileCopy_robustcheckout
gecko-t-win7-32-cu      [File]DirectoryCreate_GenericWorkerDirectory
gecko-t-win7-32-cu      [File]DirectoryCreate_LogDirectory
gecko-t-win7-32-cu      [File]DirectoryCreate_MozillaMaintenanceDir
gecko-t-win7-32-cu      [File]DirectoryCreate_SublimeText3_PackagesFolder
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_GenericWorkerDownload
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_LiveLogDownload
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_MercurialCerts
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_MercurialConfig
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_MozFakeCA_2017_10_13_cer
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_MozFakeCA_cer
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_MozRoot_cer
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_NxLogPaperTrailConfiguration
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_PaperTrailEncryptionCertificate
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_maintenanceservice
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_maintenanceservice_installer
gecko-t-win7-32-cu      [Script]ChecksumFileDownload_robustcheckout
gecko-t-win7-32-cu      [Script]CommandRun_GenericWorkerInstall
gecko-t-win7-32-cu      [Script]FirefoxBuildSecrets
gecko-t-win7-32-cu      [Script]InstallSupportingModules
gecko-t-win7-32-gpu-b   [Script]ChecksumFileDownload_MozFakeCA_2017_10_13_cer
gecko-t-win7-32-gpu-b   [Script]ChecksumFileDownload_MozFakeCA_cer
gecko-t-win7-32-gpu-b   [Script]ChecksumFileDownload_MozRoot_cer
gecko-t-win7-32-gpu-b   [Script]ChecksumFileDownload_maintenanceservice
gecko-t-win7-32-gpu-b   [Script]ChecksumFileDownload_maintenanceservice_installer

Comment 15

[Pete Moore [:pmoore][:pete]]

(the above list was compiled from scraping logs, over a selection of papertrail history)

Comment 16

[Rob Thijssen [:grenade (EET/UTC+0300)]]

working on a patch...
https://github.com/mozilla-releng/OpenCloudConfig/pull/124

Comment 17

[Rob Thijssen [:grenade (EET/UTC+0300)]]

- ChecksumFileDownload_GenericWorkerDownload
  added sha 512 checksums for 8.2.0 (32 bit), 8.3.0 (32, 64 bit), 10.6.0 (64 bit)
- ChecksumFileDownload_MozFakeCA_2017_10_13_cer, ChecksumFileDownload_MozFakeCA_cer
  modified fetch url to use gh cdn (raw.githubusercontent.com) which is unaffected by the tls change so far
- ChecksumFileDownload_MozRoot_cer
  modified fetch url to use s3 (s3.amazonaws.com/windows-opencloudconfig-packages)
- ChecksumFileDownload_maintenanceservice, ChecksumFileDownload_maintenanceservice_installer
  added sha 512 checksums and verified files in tooltool
- FileDownload_LiveLogDownload
  added sha 512 checksums for 1.1.0 (32, 64 bit)
- ZipInstall_ProcessMonitor
  vendored the artifact on s3 and tooltool because the version changed on sysinternals without the url being modified

Comment 18

[Rob Thijssen [:grenade (EET/UTC+0300)]]

i'm not sure what's going on with the directory create failures on cu. would need more info to determine if it's tls related.

Comment 19

[Rob Thijssen [:grenade (EET/UTC+0300)]]

https://github.com/mozilla-releng/OpenCloudConfig/pull/125 may help us to get tls 1.2 working in dsc.
waiting for a quiet window to merge.

Comment 20

[Rob Thijssen [:grenade (EET/UTC+0300)]]

this is now fixed. the solution was to add TLS 1.2 support like so:

>  [Net.ServicePointManager]::SecurityProtocol = ([Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12)

and to set registry keys that ensure ALL .net applications use strong cryptography by default so that the setting is also picked up by the dsc scheduled task.