1285849 - hsts/hpkp/blocklist update broken on mozilla-esr45 since 2016-06-02

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Bug 1253757 enabled release promotion on esr45, disabling nightlies so we don't have any binaries to download, eg the latest attempt 

INFO: wget -nv --no-check-certificate http://archive.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr45/firefox-45.2.1.en-US.linux-x86_64.tar.bz2
http://archive.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr45/firefox-45.2.1.en-US.linux-x86_64.tar.bz2:
2016-07-09 03:02:47 ERROR 404: Not Found.

(http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-esr45-linux64/mozilla-esr45-linux64-periodicupdate-bm74-build1-build3.txt.gz)

We get something much more powerful to use instead though - the taskcluster index. So we can query for the latest taskID with

https://index.taskcluster.net/v1/task/gecko.v2.mozilla-esr45.latest.firefox.linux64-opt

yielding EDLDA3BeT6WVjlP3a27Isg. Then request the artifacts for that taskID with

https://queue.taskcluster.net/v1/task/EDLDA3BeT6WVjlP3a27Isg/artifacts

We need to keep the old way of getting binaries going for Thunderbird, which doesn't upload anything to taskcluster.

Comment 1

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Nick, is there a status update on this, and/or anything I can do to help move this forward? The hsts preload list hasn't been updated in esr45 for >90 days, which is not great.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Teach script about the taskcluster index

Still testing this.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

This was the result of running the script - http://hg.mozilla.org/releases/mozilla-esr45/pushloghtml?changeset=6cd3802d5eda
Went ahead and pushed it since we'll be starting 45.4.0esr soon.

TODO - tidy up exit code when taskId can't be looked up; add --use-taskcluster arg to buildbot for Firefox (but not Thunderbird).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Teach script about the taskcluster index v2

Makes using the taskcluster index the default, and adds --use-ftp-builds so Thunderbird can keep using archive.m.o.

Also 
* drops the old hg bundle and mirrors in favour of the ones the hg server publishes. hgtool and HG_SHARE_BASE_DIR handle sharing existing checkouts
* uses https, since our wget supports hg.m.o, archive.m.o, and the tc endpoints
* drops tinderbox-builds/${REPODIR}-${TBOX_BUILDS_PLATFORM}/latest because we no longer update that
* misc comment tidy up

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Pass --use-ftp-builds to script

Also drops hg bundle/mirror args.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Set use_ftp_for_xpcshell for Thundbird

No taskcluster index available for TB :-(

Comment 7

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8789113 [details] [diff] [review]
[buildbotcustom] Pass --use-ftp-builds to script

Review of attachment 8789113 [details] [diff] [review]:
-----------------------------------------------------------------

::: misc.py
@@ +2363,5 @@
>      if config['enable_hsts_update'] is True:
>          extra_args.extend(['--hsts'])
>      if config['enable_hpkp_update'] is True:
>          extra_args.extend(['--hpkp'])
> +    if config.get('use_ftp_for_xpcshell') is True:

If we're using config.get(), should we add a default of False in there?

Comment 8

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8789112 [details] [diff] [review]
[tools] Teach script about the taskcluster index v2

Review of attachment 8789112 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/periodic_file_updates/periodic_file_updates.sh
@@ +107,5 @@
>  
>      cd "${BASEDIR}"
>      VERSION_URL_HG="${VERSION_REPO}/raw-file/default/${APP_DIR}/config/version.txt"
>      rm -f ${VERSION_FILE}
> +    ${WGET} -O ${VERSION_FILE} ${VERSION_URL_HG}

Have we verified that we do properly check the certificate now?

@@ +169,5 @@
> +        exit 22
> +    else
> +        echo "INFO: Got taskId of $TASKID"
> +    fi
> +    # hack! really want the last run, which may not be the first (zeroth)

Well at least we're calling it out! ;)

@@ -406,5 @@
>          if [ -f "${HGTOOL}" ]; then
>              # Need to pass the default branch here to avoid pollution from buildprops.json
>              # when hgtool.py is run in production.
>              CLONE_CMD="${HGTOOL} --branch default"
> -            if [ "${MIRROR}" != "" ]; then

Yay for ripping this out.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8789113 [details] [diff] [review]
[buildbotcustom] Pass --use-ftp-builds to script

With a default of False:
https://hg.mozilla.org/build/buildbotcustom/rev/866dceafb8d3
https://hg.mozilla.org/build/buildbotcustom/rev/2bcf33f2977a

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8789114 [details] [diff] [review]
[buildbot-configs] Set use_ftp_for_xpcshell for Thundbird

https://hg.mozilla.org/build/buildbot-configs/rev/36d8669c5156
https://hg.mozilla.org/build/buildbot-configs/rev/c60b43fa9e07

I've verified we get no cert error using https without --no-check-certificate, but haven't checked if wget is doing a python < 2.7.latest.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8789112 [details] [diff] [review]
[tools] Teach script about the taskcluster index v2

https://hg.mozilla.org/build/tools/rev/39f129de3405

I'll verify the state of all the branches after the weekend's jobs.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

m-c:      http://hg.mozilla.org/mozilla-central/pushloghtml?changeset=6138e059bacb
m-a:      http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?changeset=66754db69a70
m-b:      failed to update blocklist, but only because the tree is closed; script was fine
m-r :     https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=4438b2af5ef0
m-esr45:  http://hg.mozilla.org/releases/mozilla-esr45/pushloghtml?changeset=db9a925327df

c-c:      https://hg.mozilla.org/comm-central/rev/428961bc2465fdbac9c20ceac6e66c0a62e4a86a
c-a:      https://hg.mozilla.org/releases/comm-aurora/rev/f8eeabb787338f6e9d0b8d28260475f8ed56885f
c-b:      https://hg.mozilla.org/releases/comm-beta/rev/00e2fcfe8dfb772f9453d0a717789e9e99dba624
c-esr45:  https://hg.mozilla.org/releases/comm-esr45/rev/36bcb0a40b6fe69c6a3b78fed18531a254bbf229

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

m-r should be https://hg.mozilla.org/releases/mozilla-release/rev/4438b2af5ef058c9fb2d55c9e3dded58b32797c6

Comment 14

[Daniel Veditz [:dveditz]]

Looks like the previous built-in preload list expired Sept 3. For sites which don't emit a HPKP header (like addons.mozilla.org!) ESR users were unprotected from MITM from that date until they get the new release. That was originally scheduled for Sept 13 but is now Sept 20.

Comment 15

[Daniel Veditz [:dveditz]]

This should have been flagged as a security problem for ESR releases. We removed our hand-rolled cert pinning for add-on updates on the theory that HPKP replaced that protection mechanism.

Comment 16

[Liz Henry (:lizzard) (relman/hg->git project)]

It looks like we changed the date for the expiration to Dec. 17th the day before we built the candidate for ESR45.4.0,  http://hg.mozilla.org/releases/mozilla-esr45/rev/55e768767416   Dan and I discussed this on Friday or Saturday and didn't think we needed to rebuild ESR. Noting this as the question came up Tues. morning again.