861960 - HSTS preload list script: factor out obtaining the seed list

Status: RESOLVED WONTFIX

(Core :: Security: PSM, defect)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Google recently changed the web interface to Chrome's code, and it is not currently possible to get a raw source file over http(s). Consequently, the HSTS preload list script fails when it attempts to get the seed list. It's increasingly obvious that we can't rely on that file being available or even in a format we expect. So, I propose we refactor that step out of building our preload list. That is, instead of fetching and parsing some remote file for the seed list, we supply a local file we know is good. This way, we can actually expand our preload list beyond what is in Chrome's list if/when we decide to do so. For right now, we would probably just run `svn export https://src.chromium.org/chrome/trunk/src/net/http/transport_security_state_static.json' and then use the checked out file.

Comment 1

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Actually, I take half of that back: is looks like the list is available as a raw file at https://src.chromium.org/chrome/trunk/src/net/http/transport_security_state_static.json. However, the point still stands that we can't rely on this always being available at this location (or any location we don't control) and in a format we expect.

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

FWIW, I agree.

Comment 3

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: patch

The one thing I'm concerned about is that this changes how the script is called, which requires changes in the automation infrastructure. I could hard-code our two expected input files, but I'm not sure that's the best solution.

Comment 4

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 762811 [details] [diff] [review]
patch

Review of attachment 762811 [details] [diff] [review]:
-----------------------------------------------------------------

Camilo, can you trade me this review for reviews of your insanity patches?

Comment 5

[Camilo Viecco (:cviecco)]

Comment on attachment 762811 [details] [diff] [review]
patch

Review of attachment 762811 [details] [diff] [review]:
-----------------------------------------------------------------

However I would put in some defaults (ie, do we need this patch?)

Comment 6

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: patch v1.1

Thanks for the review. I tried to think of a good way to be able to just use defaults in the script, but for the Mozilla list at least, I think it's important to be able to have a local seed file, and I don't think it's a good idea to hard-code the path to that file. So, passed-in parameters it is.

I made some changes to guard against duplicates, but nothing functional changed.
Carrying over reviews.

Comment 7

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

coop - this will require changes to how the script is run. In particular, it will need to check out a copy of the hsts_preload_candidates.json file this patch adds. Then, it just needs to pass "https://src.chromium.org/chrome/trunk/src/net/http/transport_security_state_static.json" and the absolute path to hsts_preload_candidates.json as additional arguments to the script. Is this reasonable? We might also want to go back to doing dry runs until we're confident this does the right thing. Thanks!

Comment 8

[Chris Cooper [:coop] (he/him)]

Existing script is here:

https://hg.mozilla.org/build/tools/file/db090f1d4c67/scripts/hsts/update_hsts_preload_list.sh

Could easily jerry-rig it to update a user repo for manual testing (possibly just by setting the BRANCH) until you're satisfied it's working correctly.

I'm away right now, so won't have a chance to look at this myself until late next week.

Comment 9

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

What we have is working for now.