Closed Bug 1435729 Opened 6 years ago Closed 6 years ago

Update packages on OSX to support TLS > 1.0


(Infrastructure & Operations :: RelOps: General, task)

Not set


(Not tracked)



(Reporter: fubar, Assigned: dhouse)




(1 file, 1 obsolete file)

We need to build new custom packages on OS X to support recent versions of TLS (ie 1.2). I expect we'll need: openssl, python, curl, and wget to start. We'll need versions for both 10.7 and 10.10.

Github will be disabling access from non-TLS1.2 clients later this month (Feb 22), with a one hour test this Thurs (Feb 8) (bug 1435441), and NSM scans show that all of the OSX builders/testers will fail (bug 1435341).
Jake, I'd like to do this to learn what is involved. I'll get a loaner and I'll ping you for reviews, and help :)
Assignee: relops → dhouse
Depends on: 1435734
In puppet, we have a module "git::repo" that uses the git commandline tool directly to clone repositories. This module is not used on the OSX machines: only used for scriptworkers (signing/pushapk/balrog/shipit/beetmover/transparency: mozilla-releng/cot-gpg-keys), cruncher (catlee/reporter), aws_manager/slaveapi (mozilla/build-cloud-tools), and roller (mozilla-platform-ops/relops-hardware-controller).
Checking papertrail for activity at the times recorded by NSM scan for TLS <1.2 connections (from bug 1435341), I see tooltool being pulled.
For example, from at 18:24 UTC Feb 02:
Feb 02 18:24:12 generic-worker: 2018/02/02 10:24:12 Running task
10:24:38     INFO - Downloading to /Users/cltbld/tasks/task_1517590499/build/
notes for myself on the data collection:
# dump a log from nsm into a file, then look up the hostnames from the ips
cat tls1.txt |jq -r '.["id.orig_h"]'|sort|uniq|xargs -I {} ssh 'echo "{} $(dig +short -x {})"' >> names
# for each name, get the times reported and look up around those time in papertrail
off=1; (while read ip name; do if [[ $name = *"yosemite"* ]]; then echo $ip; export TZ=UTC; grep $ip tls1.txt|jq -r '.["ts"] | strftime("%Y-%m-%d %R %Z")'|sed 's/\n/ /g'|(while read ts; do echo "$ts ${name%\.*} "; start=\"$(export TZ=UTC; date -d "$ts - $off minutes" +'%Y-%m-%d %R %Z')\"; end=\"$(export TZ=UTC; date -d "$ts + $off minutes" +'%Y-%m-%d %R %Z')\"; papertrail --min-time "$start" --max-time "$end" "${name%\.*}"; done)>>"tls_${name}"; fi; done <names
I am curious if we need to be concerned about the https pulls from rawgithub. According to the github engineering blog notice, they will be disabling on and the api domain: "TLSv1/TLSv1.1: This applies to all HTTPS connections, including web, API, and git connections to and" (
I sent an inquiry to github support asking if the "" https pulls will also be affected.
Hi Jordan, do you know if we could switch mozharness to use the /python/mozbuild/mozbuild/action/ instead of getting it from github? 

I'm considering this as a stop gap for the github TLS <1.2 block. And upgrading openssl and python on OSX may take longer than the deadline.

A. There is a 2015 copy of the file in puppet and is placed into /builds/ on all non-windows systems. We could update that and use it.

B. Better long-term: Pull it from hg.m.o instead?
1. The default baseurl is different.
2. The hgmo version is newer. The github one was last updated in '16 (, and on hgmo in Nov '17 (

$ diff
< #!/usr/bin/env python
> #!/usr/bin/env python2.7
<         if self.validate_size():
>         if self.size is None or self.validate_size():
<         print "%s\t%s\t%s" % ("P" if f.present() else "-",
>         print("%s\t%s\t%s" % ("P" if f.present() else "-",
<                               f.filename)
>                               f.filename))
<                         os.makedirs(cache_folder, 0700)
>                         os.makedirs(cache_folder, 0o0700)
<         options_obj.base_url = ['']
>         options_obj.base_url = ['']

$ host is an alias for is an alias for has address has address has address
$ host is an alias for has address

default was changed in
"Rok Garbas - Bug 1284475 - migrate ToolTool blueprint to new codebase of relengapi r=KWierso"
Flags: needinfo?(jlund)
Github support responded that the raw githubuser files will still be available via TLS <1.2, but not forever. The "may" wording is unclear, and I've asked for clarification.

Date: Wed, 07 Feb 2018 13:18:11 +0000 (UTC)
From: "Stacey Burns (GitHub Staff)" <>
To: David House <>
Subject: Re: Crypto removal notice

Hi David,

You may still be able to access raw downloads for a period of time but this may be something that we disable in the future/near future, so we wouldn't be able to make any guarantees. 


> Will raw file downloads via TLS 1.0 be disabled?
> Specifically, will I still be able to pull with TLS 1.0?
> Thank you
Greg, can we download from hgmo instead of github? I think github is not turning off TLS <1.2 for, but this could work-around it if they do. And this is the only TLS<1.2 github connection I've found on the osx machines.

I'm hoping and guessing that this will get pulled in as mozilla-central/testing/mozharness/mozharness/mozilla/ (do you know?)
Attachment #8949183 - Flags: review?(jlund)
Attachment #8949183 - Flags: review?(gps)
Comment on attachment 8949183 [details] [diff] [review]
pull from hgmo not githubusercontent

Review of attachment 8949183 [details] [diff] [review]:

I have a better idea. Will submit a patch shortly.
Attachment #8949183 - Flags: review?(gps) → review-
Comment on attachment 8949183 [details] [diff] [review]
pull from hgmo not githubusercontent

Review of attachment 8949183 [details] [diff] [review]:

I should have said why this is r-. The main reason is lack of determinism. If we ever delete python/mozbuild/mozbuild/action/ from mozilla-central, code in the wild will stop working.

This was a problem before. But mozilla-central is probably a bit more dynamic than mozilla/build-tooltool on GitHub.

Anyway, I submitted a new patch that has mozharness always use the copy of from inside the repo. This eliminates the potential for a MitM attack compromising our download of And it should make CI more reliable, as we won't depend on any external service to obtain if we have a copy of mozharness we have a copy of
Attachment #8949183 - Attachment is obsolete: true
Attachment #8949183 - Flags: review?(jlund)
Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored;

:gps thank you. This is much better.
Github support responded to me again and confirmed that is not included the TLS restriction. But they warned that we shouldn't count on it not also changing in the future.
Comment on attachment 8949183 [details] [diff] [review]
pull from hgmo not githubusercontent

Review of attachment 8949183 [details] [diff] [review]:

::: mozharness/mozilla/
@@ +14,2 @@
> +    ""

hm, looks like there are differences between these clients:

--- /Users/jlund/Downloads/   2018-02-07 15:37:15.000000000 -0800
+++ /Users/jlund/Downloads// 2018-02-07 15:36:52.000000000 -0800
@@ -1,6 +1,6 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2.7

 # tooltool is a lookaside cache implemented in Python
 # Copyright (C) 2011 John H. Ford <>
 # This program is free software; you can redistribute it and/or
@@ -139,11 +139,11 @@ class FileRecord(object):
                 "trying to validate digest on a missing file, %s', self.filename")
             raise MissingFileException(filename=self.filename)

     def validate(self):
-        if self.validate_size():
+        if self.size is None or self.validate_size():
             if self.validate_digest():
                 return True
         return False

     def describe(self):
@@ -370,13 +370,13 @@ def list_manifest(manifest_file):
         return False
     for f in manifest.file_records:
-        print "%s\t%s\t%s" % ("P" if f.present() else "-",
+        print("%s\t%s\t%s" % ("P" if f.present() else "-",
                               "V" if f.present() and f.validate() else "-",
-                              f.filename)
+                              f.filename))
     return True

 def validate_manifest(manifest_file):
     """I validate that all files in a manifest are present and valid but
@@ -672,11 +672,11 @@ def fetch_files(manifest_file, base_urls
             if cache_folder:
       "Updating local cache %s..." % cache_folder)
                     if not os.path.exists(cache_folder):
               "Creating cache in %s..." % cache_folder)
-                        os.makedirs(cache_folder, 0700)
+                        os.makedirs(cache_folder, 0o0700)
                     shutil.copy(os.path.join(os.getcwd(), localfile.filename),
                                 os.path.join(cache_folder, localfile.digest))
           "Local cache %s updated with %s" % (cache_folder,
                     touch(os.path.join(cache_folder, localfile.digest))
@@ -997,11 +997,11 @@ def main(argv, _skip_logging=False):

     (options_obj, args) = parser.parse_args(argv[1:])

     # default the options list if not provided
     if not options_obj.base_url:
-        options_obj.base_url = ['']
+        options_obj.base_url = ['']

     # ensure all URLs have a trailing slash
     def add_slash(url):
         return url if url.endswith('/') else (url + '/')
     options_obj.base_url = [add_slash(u) for u in options_obj.base_url]

Rok has recently switched a lot of tooltool calls to the new service:

I'm concerned why this wasn't done here and what this may break.
nevermind, we have a new patch. Looking now.
Flags: needinfo?(jlund)
Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored;

In general, I really like this patch. I think part of what this solves is also the thing that may cause bustage: differences between contents out in the wild (read, used in production).

Looks like what I said in: still holds true for this patch. Given that, I think it's worth noting that some (many?) instances where we use tooltool, will be different after today. Namely, the server they reach: vs

I think that's okay and it would be good to weed out the instances that are not working. Have you run this on try? Some of these are release related changes (single locales) so we should uplift to beta once it sticks on central and catch errors sooner rather than later on the next beta.
Attachment #8949195 - Flags: review?(jlund) → review+
Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored; makes me think this patch is okay to do. It may have been missed when we made the tooltool switch so this could actually be more helpful than we realize.
Comment on attachment 8949195 [details]
Bug 1435729 - Always use vendored;

If there are problems with the copy of in tree, we should fix the in-tree copy.

I have noticed there have been patches to the code in python/mozbuild/mozbuild/, which is where the mozharness was copied from. I think it's best to keep in the in-repo files in sync. Some of those changes (such as the octal syntax change to appease Python 3) should be upstreamed.
Pushed by
Always use vendored; r=jlund
Blocks: 1436612
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 1435441
You need to log in before you can comment on or make changes to this bug.