1093295 - HSTS and HPKP automatic updates are possibly broken on B2G release branches

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

It looks like the HSTS (and HPKP, for 34) automatic update scripts are broken somehow on B2g release branches (30, 32, and 34):

09:24 RyanVM|sheriffduty | so I could have sworn we were doing automated HSTS updates on the B2G release branches, but apparently we aren't
09:25                  * | RyanVM|sheriffduty had to push a manual update to b2g30 today due to xpcshell going permafail
09:25             keeler | I could have sworn we were doing them as well...
09:26             keeler | I seem to recall something about when a new branch was made, there was something someone had to do to include it in the list of
                         | branches that automatically did that?
09:28 RyanVM|sheriffduty | there's a flag for it in the buildbot configs
09:28             keeler | ah
09:29 RyanVM|sheriffduty | but they usually copy/paste from the older ones
...
09:32 RyanVM|sheriffduty | so it appears that the branches are configured to do automatic hsts updates
09:32 RyanVM|sheriffduty | which means they're broken
09:32 RyanVM|sheriffduty | not misconfigured
09:32 RyanVM|sheriffduty | "yay"
09:32             keeler | :(
...
09:32 RyanVM|sheriffduty | it's all active b2g branches
09:32 RyanVM|sheriffduty | b2g30, b2g32, b2g34
09:32 RyanVM|sheriffduty | I did a manual update on all 3 this morning

It may be the case that the patch from bug 1083085 needs to be adapted for these branches (and, indeed, all long-term branches that need this information to be fresh). However, looking at the changelogs, that doesn't appear to be the only thing that's wrong here.

Comment 1

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

:coop, is this something you can shed some light on? Are there logs somewhere that would tell us what's going on?

Comment 2

[Phil Ringnalda (:philor)]

e.g. http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-b2g30_v1_4-linux64/mozilla-b2g30_v1_4-linux64-periodicupdate-bm70-build1-build0.txt.gz

The job downloads the latest linux64 Firefox nightly for the branch, and then... there is no and then, because there are no desktop Firefox nightlies on b2g branches, only on-push builds.

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to David Keeler (:keeler) [use needinfo?] from comment #1)
> :coop, is this something you can shed some light on? Are there logs
> somewhere that would tell us what's going on?

As pilor says, the current script relies on an in-branch linux64 build to grab the js engine used to run the update script. We could avoid this if by using a static version of the engine.

Does the version of the js engine matter at all?

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to Chris Cooper [:coop] [away until Nov 17] from comment #3)
> Does the version of the js engine matter at all?

For the HSTS updater at least, it should probably be from a recent aurora or nightly build, because we do rely on things like nsISiteSecurityService to parse the headers we see.

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to David Keeler (:keeler) [use needinfo?] from comment #4)
> (In reply to Chris Cooper [:coop] [away until Nov 17] from comment #3)
> > Does the version of the js engine matter at all?
> 
> For the HSTS updater at least, it should probably be from a recent aurora or
> nightly build, because we do rely on things like nsISiteSecurityService to
> parse the headers we see.

Would the most recent m-c nightly be acceptable? That can be grabbed from a standard location:

http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/

Comment 6

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

That should work, yes.

Comment 7

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Chris, can you go ahead and make the change to use the most recently mozilla-central nightly available? Thanks.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #7)
> Chris, can you go ahead and make the change to use the most recently
> mozilla-central nightly available? Thanks.

I'll test something in staging today.

Comment 9

[Chris Cooper [:coop] (he/him)]

Attachment: Use mozilla-central to check for HSTS and HPKP updates

A lot of the bulk change here is from untabifying. 

The two substantive changes are:

1) Don't run with -e. This was causing us to exit out when the various diffs returned 1 (which is expected when there are differences), so we would never actually update(!).

2) Change get_version to take a branch argument so we can look up a separate version for mozilla-central and then use mozilla-central build & test packages to check for updates. This allows us to use the most-recent code to check for updates, rather than relying on code that could be up to 50 weeks out-of-date for ESR.

Comment 10

[Justin Wood (:Callek)]

Comment on attachment 8546007 [details] [diff] [review]
Use mozilla-central to check for HSTS and HPKP updates

Review of attachment 8546007 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/periodic_file_updates/periodic_file_updates.sh
@@ +99,5 @@
>      cd "${BASEDIR}"
> +    echo "INFO: Retrieving current version from ${VERSION_BRANCH}..."
> +
> +    VERSION_URL_HG="${VERSION_REPO}/raw-file/default/${APP_DIR}/config/version.txt"
> +    rm -f mc_version.txt

${VERSION_FILE} here, to match the -O from wget

Comment 11

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8546007 [details] [diff] [review]
Use mozilla-central to check for HSTS and HPKP updates

Review of attachment 8546007 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/tools/rev/605a89535550

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Interestingly, the HPKP/HSTS updates ran on b2g32 over the weekend, but not b2g34 or b2g30.

Comment 13

[Chris Cooper [:coop] (he/him)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #12)
> Interestingly, the HPKP/HSTS updates ran on b2g32 over the weekend, but not
> b2g34 or b2g30.

Filed as bug 1123965.

Comment 14

[Chris Cooper [:coop] (he/him)]

Fixed in bug 1123965.