All users were logged out of Bugzilla on October 13th, 2018

HPKP update failures across trees

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
4 years ago
5 months ago

People

(Reporter: nthomas, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
WG9s noticed no checkins this weekend to update blocklist and hsts, which looks like a failure in HPKP so no push to the repo.

eg
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm85-build1-build0.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-aurora-linux64/mozilla-aurora-linux64-periodicupdate-bm72-build1-build0.txt.gz

INFO: New HSTS preload list differs from what is in-tree.
INFO: Downloading all the necessary pieces to update HPKP...
2014-07-05 03:06:29 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/tools/genHPKPStaticPins.js [19329/19329] -> "genHPKPStaticPins.js" [1]
2014-07-05 03:06:30 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/tools/PreloadedHPKPins.json [8574/8574] -> "PreloadedHPKPins.json" [1]
2014-07-05 03:06:30 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/ssl/tests/unit/tlsserver/default-ee.der [639/639] -> "default-ee.der" [1]
2014-07-05 03:06:31 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/boot/src/StaticHPKPins.h [49816/49816] -> "StaticHPKPins.h" [1]
2014-07-05 03:06:32 URL:http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/security/manager/boot/src/StaticHPKPins.errors [3025/3025] -> "StaticHPKPins.errors" [1]
INFO: Generating new HPKP preload list...
INFO: Checking whether new HPKP preload list is valid...
StaticHPKPins.h is empty. That's less good.
program finished with exit code 52

Updated

4 years ago
Duplicate of this bug: 1034963
(Assignee)

Comment 2

4 years ago
I'll do some first-pass debugging here. This worked fine on m-c earlier in the week, so I'm wondering whether the PHX outage interrupted something here.
Assignee: nobody → coop
Status: NEW → ASSIGNED
Priority: -- → P2
Can set -x be turned on so that things are easier to debug? Also, does periodic_file_updates.sh use full path names (that's required for the genHPKPStaticPins.js generator at least)
(Assignee)

Comment 4

4 years ago
It's running the command correctly, but no output is being generated, modulo the errors:

https://coop.pastebin.mozilla.org/5531395

I don't know enough about xpcshell or the internals of the HPKP to know where this is failing.

Monica: have you tried running the same xpcshell script locally to diagnose?
Flags: needinfo?(mmc)
Hi coop,

In a case of bad timing, https://bugzilla.mozilla.org/show_bug.cgi?id=1029561 broke the pinset generator, which requires that we don't specify certs that are not builtin. This bug removed some of our builtins that Google is relying on. I'm working on a fix right now, but it will require uplifts to Aurora.

Thanks,
Monica
Flags: needinfo?(mmc)
FYI the root cert changes broke this on 7/4's nightly, right after I checked in the last change on 7/3.
(Assignee)

Comment 7

4 years ago
This is working now. Monica has already uplifted the change to aurora:

https://hg.mozilla.org/releases/mozilla-aurora/rev/d9c3d923cb3e
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.