1123965 - periodic file updates not triggering automatically on b2g30 and b2g34 branches

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

This is a spin-off of Ryan's comment in https://bugzilla.mozilla.org/show_bug.cgi?id=1093295#c12

HSTS and HPKP periodic file updates are not running on the b2g30 and b2g34 branches (and possibly others). Updates *are* running on b2g32.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

I actually discussed this with dkeeler on IRC a bit today. He thinks there's two different issues at play.

For b2g34, we need an update to the generator script. He already fixed it on b2g32 as part of fixing my earlier attempt at doing a manual update, which is why the automated updates are working correctly there.

For b2g30, he filed bug 1123823, which needs further investigation.

Comment 2

[Dana Keeler (she/her) (use needinfo) [:keeler]]

To elaborate more on the generator script issue, here's what I think is going on: in bug 1093295, we switched to using the most recently available nightly (i.e. mozilla-central) xpcshell to run the xpcshell scripts that generate the files we want to update. However, we've made changes to some of the interfaces those scripts use (e.g. nsISiteSecurityService.processHeader now takes an nsISSLStatus when it didn't before). As a result, the scripts are failing because they're only compatible with an older version of xpcshell. We've put ourselves in a difficult position in that every time there's a change in mozilla-central that affects these scripts, we will need to update them on all branches still doing these automatic updates.
I think there are two reasonable approaches to fixing this: 1. go back to using branch-specific binaries (but we have to fix this on branches where there isn't just a "nightly" or "latest" available) or 2. always use the copies of the scripts that are in mozilla-central, regardless of which branch we're updating.

(For reference, the scripts in question are security/manager/tools/genHPKPStaticPins.js and security/manager/tools/getHSTSPreloadList.js)

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #1)
> I actually discussed this with dkeeler on IRC a bit today. He thinks there's
> two different issues at play.
> 
> For b2g34, we need an update to the generator script. He already fixed it on
> b2g32 as part of fixing my earlier attempt at doing a manual update, which
> is why the automated updates are working correctly there.
 
> For b2g30, he filed bug 1123823, which needs further investigation.

Something else is still broken here. I just fixed bug 1123823 and re-ran the periodic file update for b2g30 and hit the following error:

INFO: Downloading all the necessary pieces to update HPKP...
INFO: wget -nv --no-check-certificate http://hg.mozilla.org/releases/mozilla-b2g30_v1_4/raw-file/default/security/manager/tools/genHPKPStaticPins.js
http://hg.mozilla.org/releases/mozilla-b2g30_v1_4/raw-file/default/security/manager/tools/genHPKPStaticPins.js:
2015-01-20 15:53:12 ERROR 404: Not Found.
ERROR: wget exited with a non-zero exit code: 8

Comparing b2g32 (working) vs b2g30 (not working), it looks like we're just missing the HPKP files on b2g30.

http://hg.mozilla.org/releases/mozilla-b2g32_v2_0/file/8ea706019b32/security/manager/tools
http://hg.mozilla.org/releases/mozilla-b2g30_v1_4/file/76e172ece6de/security/manager/tools

:keeler - are those files safe to add to b2g30? Do they need any branch-specific tweaks?

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #2)
> (For reference, the scripts in question are
> security/manager/tools/genHPKPStaticPins.js and
> security/manager/tools/getHSTSPreloadList.js)

Yes, these are the ones that are missing on b2g30.

Comment 5

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Oh, actually, pinning (HPKP) isn't even a thing on b2g30, so we shouldn't be trying to update those files - just the HSTS-related ones.

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #5)
> Oh, actually, pinning (HPKP) isn't even a thing on b2g30, so we shouldn't be
> trying to update those files - just the HSTS-related ones.

OK, I'll disable that.

Comment 7

[Chris Cooper [:coop] (he/him)]

Attachment: Disable HPKP update on b2g30

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #2)
> I think there are two reasonable approaches to fixing this: 1. go back to
> using branch-specific binaries (but we have to fix this on branches where
> there isn't just a "nightly" or "latest" available) or 2. always use the
> copies of the scripts that are in mozilla-central, regardless of which
> branch we're updating.

Do you have a preference? I favor reverting to using per-branch binaries.

Comment 9

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8552804 [details] [diff] [review]
Disable HPKP update on b2g30

Review of attachment 8552804 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbot-configs/rev/870b748db5e8

Comment 10

[Dana Keeler (she/her) (use needinfo) [:keeler]]

(In reply to Chris Cooper [:coop] from comment #8)
> (In reply to David Keeler [:keeler] (use needinfo?) from comment #2)
> > I think there are two reasonable approaches to fixing this: 1. go back to
> > using branch-specific binaries (but we have to fix this on branches where
> > there isn't just a "nightly" or "latest" available) or 2. always use the
> > copies of the scripts that are in mozilla-central, regardless of which
> > branch we're updating.
> 
> Do you have a preference? I favor reverting to using per-branch binaries.

If we can make that work, that would probably be best.

Comment 11

[Jordan Lund (:jlund)]

buildbot-config patch(es) are live in production

Comment 12

[Chris Cooper [:coop] (he/him)]

Attachment: Don't use mozilla-central by default, but keep it as an option

We're going to back to using per-branch binaries, but rather than rip the "use mozilla-central" code out, I've left it as an option. This will be useful if we ever need to update a branch (e.g., mozilla-beta) that doesn't have latest-* builds.

I also simplified how the versions are parsed.

Comment 13

[Chris Cooper [:coop] (he/him)]

Attachment: Don't use mozilla-central by default, but keep it as an option, v2

Same as previous patch, but remove extraneous arg from get_version().

Comment 14

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8555435 [details] [diff] [review]
Don't use mozilla-central by default, but keep it as an option, v2

Review of attachment 8555435 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/tools/rev/7046772044d4

Comment 15

[Chris Cooper [:coop] (he/him)]

Attachment: Look for download artifacts in nightly/ and tinderbox-builds/

OK, this one should fix the issues.

The script now checks first for build and test packages under the nightly/ dir (as previously), but then also tries to find a latest build under the tinderbox-builds/ dir.

Comment 16

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8557187 [details] [diff] [review]
Look for download artifacts in nightly/ and tinderbox-builds/

https://hg.mozilla.org/build/tools/rev/e003fc53b55f

Comment 17

[Chris Cooper [:coop] (he/him)]

I just witnessed successful periodic file updates on both b2g30 and b2g34.