1035003 - HPKP update failures across trees

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

WG9s noticed no checkins this weekend to update blocklist and hsts, which looks like a failure in HPKP so no push to the repo.

eg
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm85-build1-build0.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-aurora-linux64/mozilla-aurora-linux64-periodicupdate-bm72-build1-build0.txt.gz

INFO: New HSTS preload list differs from what is in-tree.
INFO: Downloading all the necessary pieces to update HPKP...
2014-07-05 03:06:29 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/tools/genHPKPStaticPins.js [19329/19329] -> "genHPKPStaticPins.js" [1]
2014-07-05 03:06:30 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/tools/PreloadedHPKPins.json [8574/8574] -> "PreloadedHPKPins.json" [1]
2014-07-05 03:06:30 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/ssl/tests/unit/tlsserver/default-ee.der [639/639] -> "default-ee.der" [1]
2014-07-05 03:06:31 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/boot/src/StaticHPKPins.h [49816/49816] -> "StaticHPKPins.h" [1]
2014-07-05 03:06:32 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/boot/src/StaticHPKPins.errors [3025/3025] -> "StaticHPKPins.errors" [1]
INFO: Generating new HPKP preload list...
INFO: Checking whether new HPKP preload list is valid...
StaticHPKPins.h is empty. That's less good.
program finished with exit code 52

Comment 2

[Chris Cooper [:coop] (he/him)]

I'll do some first-pass debugging here. This worked fine on m-c earlier in the week, so I'm wondering whether the PHX outage interrupted something here.

Comment 3

[[:mmc] Monica Chew (no longer reading bugmail)]

Can set -x be turned on so that things are easier to debug? Also, does periodic_file_updates.sh use full path names (that's required for the genHPKPStaticPins.js generator at least)

Comment 4

[Chris Cooper [:coop] (he/him)]

It's running the command correctly, but no output is being generated, modulo the errors:

https://coop.pastebin.mozilla.org/5531395

I don't know enough about xpcshell or the internals of the HPKP to know where this is failing.

Monica: have you tried running the same xpcshell script locally to diagnose?

Comment 5

[[:mmc] Monica Chew (no longer reading bugmail)]

Hi coop,

In a case of bad timing, https://bugzilla.mozilla.org/show_bug.cgi?id=1029561 broke the pinset generator, which requires that we don't specify certs that are not builtin. This bug removed some of our builtins that Google is relying on. I'm working on a fix right now, but it will require uplifts to Aurora.

Thanks,
Monica

Comment 6

[[:mmc] Monica Chew (no longer reading bugmail)]

FYI the root cert changes broke this on 7/4's nightly, right after I checked in the last change on 7/3.

Comment 7

[Chris Cooper [:coop] (he/him)]

This is working now. Monica has already uplifted the change to aurora:

https://hg.mozilla.org/releases/mozilla-aurora/rev/d9c3d923cb3e