1435729 - Update packages on OSX to support TLS > 1.0

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Kendall Libby [:fubar] (he/him)]

We need to build new custom packages on OS X to support recent versions of TLS (ie 1.2). I expect we'll need: openssl, python, curl, and wget to start. We'll need versions for both 10.7 and 10.10.

Github will be disabling access from non-TLS1.2 clients later this month (Feb 22), with a one hour test this Thurs (Feb 8) (bug 1435441), and NSM scans show that all of the OSX builders/testers will fail (bug 1435341).

Comment 1

[:dhouse]

Jake, I'd like to do this to learn what is involved. I'll get a loaner and I'll ping you for reviews, and help :)

Comment 2

[:dhouse]

In puppet, we have a module "git::repo" that uses the git commandline tool directly to clone repositories. This module is not used on the OSX machines: only used for scriptworkers (signing/pushapk/balrog/shipit/beetmover/transparency: mozilla-releng/cot-gpg-keys), cruncher (catlee/reporter), aws_manager/slaveapi (mozilla/build-cloud-tools), and roller (mozilla-platform-ops/relops-hardware-controller).

Comment 3

[:dhouse]

Checking papertrail for activity at the times recorded by NSM scan for TLS <1.2 connections (from bug 1435341), I see tooltool being pulled.
For example, from t-yosemite-r7-0150.test.releng.scl3.mozilla.com at 18:24 UTC Feb 02:
```
Feb 02 18:24:12 t-yosemite-r7-0150.test.releng.scl3.mozilla.com generic-worker: 2018/02/02 10:24:12 Running task https://tools.taskcluster.net/task-inspector/#NNlDdHUtQk6wuZARUQn6Kg/0
```
https://tools.taskcluster.net/groups/HIiXTpRiSUm51_OGLL0quA/tasks/NNlDdHUtQk6wuZARUQn6Kg/runs/0/logs/public%2Flogs%2Fawsy_info.log#L107
```
10:24:38     INFO - Downloading https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py to /Users/cltbld/tasks/task_1517590499/build/tooltool.py
```

Comment 4

[:dhouse]

(In reply to Dave House [:dhouse] from comment #3)
> https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py
> to /Users/cltbld/tasks/task_1517590499/build/tooltool.py

Comes from mozharness: https://searchfox.org/comm-central/source/mozilla/testing/mozharness/mozharness/mozilla/tooltool.py#15

https://hg.mozilla.org/build/mozharness/file/tip/mozharness/mozilla/tooltool.py#l17

Comment 5

[:dhouse]

notes for myself on the data collection:
```
# dump a log from nsm into a file, then look up the hostnames from the ips
cat tls1.txt |jq -r '.["id.orig_h"]'|sort|uniq|xargs -I {} ssh rejh1.srv.releng.mdc1.mozilla.com 'echo "{} $(dig +short -x {})"' >> names
# for each name, get the times reported and look up around those time in papertrail
off=1; (while read ip name; do if [[ $name = *"yosemite"* ]]; then echo $ip; export TZ=UTC; grep $ip tls1.txt|jq -r '.["ts"] | strftime("%Y-%m-%d %R %Z")'|sed 's/\n/ /g'|(while read ts; do echo "$ts ${name%\.*} "; start=\"$(export TZ=UTC; date -d "$ts - $off minutes" +'%Y-%m-%d %R %Z')\"; end=\"$(export TZ=UTC; date -d "$ts + $off minutes" +'%Y-%m-%d %R %Z')\"; papertrail --min-time "$start" --max-time "$end" "${name%\.*}"; done)>>"tls_${name}"; fi; done <names
```

Comment 6

[:dhouse]

I am curious if we need to be concerned about the https pulls from rawgithub. According to the github engineering blog notice, they will be disabling on github.com and the api domain: "TLSv1/TLSv1.1: This applies to all HTTPS connections, including web, API, and git connections to https://github.com and https://api.github.com." (https://githubengineering.com/crypto-removal-notice/)
I sent an inquiry to github support asking if the "raw.githubusercontent.com" https pulls will also be affected.

Comment 7

[:dhouse]

Hi Jordan, do you know if we could switch mozharness to use the /python/mozbuild/mozbuild/action/tooltool.py instead of getting it from github? 

I'm considering this as a stop gap for the github TLS <1.2 block. And upgrading openssl and python on OSX may take longer than the deadline.

A. There is a 2015 copy of the tooltool.py file in puppet and is placed into /builds/ on all non-windows systems. We could update that and use it.

B. Better long-term: Pull it from hg.m.o instead?
1. The default baseurl is different.
2. The hgmo version is newer. The github one was last updated in '16 (https://github.com/mozilla/build-tooltool/commits/master/tooltool.py), and on hgmo in Nov '17 (https://hg.mozilla.org/mozilla-central/log/tip/python/mozbuild/mozbuild/action/tooltool.py).

```
$ diff mozilla-cental-python-mozbuild-tooltool.py githubusercontent-tooltool.py
1c1
< #!/usr/bin/env python
---
> #!/usr/bin/env python2.7
144c144
<         if self.validate_size():
---
>         if self.size is None or self.validate_size():
375c375
<         print "%s\t%s\t%s" % ("P" if f.present() else "-",
---
>         print("%s\t%s\t%s" % ("P" if f.present() else "-",
377c377
<                               f.filename)
---
>                               f.filename))
677c677
<                         os.makedirs(cache_folder, 0700)
---
>                         os.makedirs(cache_folder, 0o0700)
1002c1002
<         options_obj.base_url = ['https://api.pub.build.mozilla.org/tooltool/']
---
>         options_obj.base_url = ['https://tooltool.mozilla-releng.net/']
```

```
$ host tooltool.mozilla-releng.net
tooltool.mozilla-releng.net is an alias for kochi-11433.herokussl.com.
kochi-11433.herokussl.com is an alias for elb097973-874527630.us-east-1.elb.amazonaws.com.
elb097973-874527630.us-east-1.elb.amazonaws.com has address 54.221.222.170
elb097973-874527630.us-east-1.elb.amazonaws.com has address 54.225.211.40
elb097973-874527630.us-east-1.elb.amazonaws.com has address 23.21.54.237
$ host api.pub.build.mozilla.org
api.pub.build.mozilla.org is an alias for relengapi-zlb.vips.scl3.mozilla.com.
relengapi-zlb.vips.scl3.mozilla.com has address 63.245.215.27
```

default was changed in https://hg.mozilla.org/mozilla-central/rev/0eebae2f0977#l4.13
"Rok Garbas - Bug 1284475 - migrate ToolTool blueprint to new codebase of relengapi r=KWierso"

Comment 8

[:dhouse]

Github support responded that the raw githubuser files will still be available via TLS <1.2, but not forever. The "may" wording is unclear, and I've asked for clarification.

```mail
Date: Wed, 07 Feb 2018 13:18:11 +0000 (UTC)
From: "Stacey Burns (GitHub Staff)" <[e-mail]>
To: David House <[e-mail]>
Subject: Re: Crypto removal notice

Hi David,

You may still be able to access raw downloads for a period of time but this may be something that we disable in the future/near future, so we wouldn't be able to make any guarantees. 

Thanks
Stacey 

> Will raw file downloads via TLS 1.0 be disabled?
> Specifically, will I still be able to pull https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py with TLS 1.0?
> 
> Thank you
```

Comment 9

[:dhouse]

Attachment: pull tooltool.py from hgmo not githubusercontent

Greg, can we download tooltool.py from hgmo instead of github? I think github is not turning off TLS <1.2 for githubusercontent.com, but this could work-around it if they do. And this is the only TLS<1.2 github connection I've found on the osx machines.

I'm hoping and guessing that this will get pulled in as mozilla-central/testing/mozharness/mozharness/mozilla/tooltool.py (do you know?)

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8949183 [details] [diff] [review]
pull tooltool.py from hgmo not githubusercontent

Review of attachment 8949183 [details] [diff] [review]:
-----------------------------------------------------------------

I have a better idea. Will submit a patch shortly.

Comment 11

[Gregory Szorc [:gps]]

Attachment: Bug 1435729 - Always use vendored tooltool.py;

Previously, we were downloading tooltool.py from random servers.
Considering tooltool.py is used to secure the download of future
components, downloading tooltool.py from potentially 3rd party
services was a major lapse in our end-to-end security, as a
compromised tooltool.py wouldn't honor integrity checks.

This commit copies the already vendored copy of tooltool.py into
the mozharness directory. A copy needs to be in the mozharness
directory because then a copy of mozharness without access to
a source checkout will have access to it.

We modify the code in mozharness that fetches tooltool to use
the copy from mozharness (unless `mach artifact toolchain` is
available).

Since a copy of tooltool.py is always reliably available, we
can remove all config entries related to tooltool.py.

Review commit: https://reviewboard.mozilla.org/r/218582/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/218582/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8949183 [details] [diff] [review]
pull tooltool.py from hgmo not githubusercontent

Review of attachment 8949183 [details] [diff] [review]:
-----------------------------------------------------------------

I should have said why this is r-. The main reason is lack of determinism. If we ever delete python/mozbuild/mozbuild/action/tooltool.py from mozilla-central, code in the wild will stop working.

This was a problem before. But mozilla-central is probably a bit more dynamic than mozilla/build-tooltool on GitHub.

Anyway, I submitted a new patch that has mozharness always use the copy of tooltool.py from inside the repo. This eliminates the potential for a MitM attack compromising our download of tooltool.py. And it should make CI more reliable, as we won't depend on any external service to obtain tooltool.py: if we have a copy of mozharness we have a copy of tooltool.py.

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored tooltool.py;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/218582/diff/1-2/

Comment 14

[:dhouse]

Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored tooltool.py;

:gps thank you. This is much better.

Comment 15

[:dhouse]

Github support responded to me again and confirmed that githubusercontent.com is not included the TLS restriction. But they warned that we shouldn't count on it not also changing in the future.

Comment 16

[Jordan Lund (:jlund)]

Comment on attachment 8949183 [details] [diff] [review]
pull tooltool.py from hgmo not githubusercontent

Review of attachment 8949183 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozharness/mozilla/tooltool.py
@@ +14,2 @@
>  TOOLTOOL_PY_URL = \
> +    "https://hg.mozilla.org/mozilla-central/raw-file/tip/python/mozbuild/mozbuild/action/tooltool.py"

hm, looks like there are differences between these clients:

--- /Users/jlund/Downloads/tooltool_github.py   2018-02-07 15:37:15.000000000 -0800
+++ /Users/jlund/Downloads//tooltool.py 2018-02-07 15:36:52.000000000 -0800
@@ -1,6 +1,6 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7

 # tooltool is a lookaside cache implemented in Python
 # Copyright (C) 2011 John H. Ford <[e-mail]>
 #
 # This program is free software; you can redistribute it and/or
@@ -139,11 +139,11 @@ class FileRecord(object):
             log.debug(
                 "trying to validate digest on a missing file, %s', self.filename")
             raise MissingFileException(filename=self.filename)

     def validate(self):
-        if self.validate_size():
+        if self.size is None or self.validate_size():
             if self.validate_digest():
                 return True
         return False

     def describe(self):
@@ -370,13 +370,13 @@ def list_manifest(manifest_file):
             manifest_file,
             str(e),
         ))
         return False
     for f in manifest.file_records:
-        print "%s\t%s\t%s" % ("P" if f.present() else "-",
+        print("%s\t%s\t%s" % ("P" if f.present() else "-",
                               "V" if f.present() and f.validate() else "-",
-                              f.filename)
+                              f.filename))
     return True


 def validate_manifest(manifest_file):
     """I validate that all files in a manifest are present and valid but
@@ -672,11 +672,11 @@ def fetch_files(manifest_file, base_urls
             if cache_folder:
                 log.info("Updating local cache %s..." % cache_folder)
                 try:
                     if not os.path.exists(cache_folder):
                         log.info("Creating cache in %s..." % cache_folder)
-                        os.makedirs(cache_folder, 0700)
+                        os.makedirs(cache_folder, 0o0700)
                     shutil.copy(os.path.join(os.getcwd(), localfile.filename),
                                 os.path.join(cache_folder, localfile.digest))
                     log.info("Local cache %s updated with %s" % (cache_folder,
                                                                  localfile.filename))
                     touch(os.path.join(cache_folder, localfile.digest))
@@ -997,11 +997,11 @@ def main(argv, _skip_logging=False):

     (options_obj, args) = parser.parse_args(argv[1:])

     # default the options list if not provided
     if not options_obj.base_url:
-        options_obj.base_url = ['https://api.pub.build.mozilla.org/tooltool/']
+        options_obj.base_url = ['https://tooltool.mozilla-releng.net/']

     # ensure all URLs have a trailing slash
     def add_slash(url):
         return url if url.endswith('/') else (url + '/')
     options_obj.base_url = [add_slash(u) for u in options_obj.base_url]



Rok has recently switched a lot of tooltool calls to the new service: https://tooltool.mozilla-releng.net/

I'm concerned why this wasn't done here and what this may break.

Comment 17

[Jordan Lund (:jlund)]

nevermind, we have a new patch. Looking now.

Comment 18

[Jordan Lund (:jlund)]

Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored tooltool.py;

https://reviewboard.mozilla.org/r/218582/#review224408

In general, I really like this patch. I think part of what this solves is also the thing that may cause bustage: differences between tooltool.py contents out in the wild (read, used in production).

Looks like what I said in: https://bugzilla.mozilla.org/show_bug.cgi?id=1435729#c16 still holds true for this patch. Given that, I think it's worth noting that some (many?) instances where we use tooltool, will be different after today. Namely, the server they reach: https://api.pub.build.mozilla.org/tooltool/ vs https://tooltool.mozilla-releng.net

I think that's okay and it would be good to weed out the instances that are not working. Have you run this on try? Some of these are release related changes (single locales) so we should uplift to beta once it sticks on central and catch errors sooner rather than later on the next beta.

Comment 19

[Jordan Lund (:jlund)]

Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored tooltool.py;

https://reviewboard.mozilla.org/r/218582/#review224412

https://hg.mozilla.org/mozilla-central/rev/0eebae2f0977 makes me think this patch is okay to do. It may have been missed when we made the tooltool switch so this could actually be more helpful than we realize.

Comment 20

[Gregory Szorc [:gps]]

Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored tooltool.py;

https://reviewboard.mozilla.org/r/218582/#review224416

If there are problems with the copy of tooltool.py in tree, we should fix the in-tree copy.

I have noticed there have been patches to the code in python/mozbuild/mozbuild/action.py, which is where the mozharness tooltool.py was copied from. I think it's best to keep in the in-repo tooltool.py files in sync. Some of those changes (such as the octal syntax change to appease Python 3) should be upstreamed.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d09848a31f54
Always use vendored tooltool.py; r=jlund

Comment 22

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/d09848a31f54