1083085 - Update source URL used for fetching HSTS and HPKP preload lists from Google

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Reed Loden [:reed]]

https://src.chromium.org/chrome/trunk/src/net/http/transport_security_state_static.json and https://src.chromium.org/chrome/trunk/src/net/http/transport_security_state_static.certs aren't getting updated anymore...

Need to figure out what the new URLs should be and update both security/manager/tools/PreloadedHPKPins.json and security/manager/tools/getHSTSPreloadList.js to use the new source URLs.

Comment 1

[Reed Loden [:reed]]

We're 7-8 weeks out-of-date right now. :( -- this should be fixed and backported to the various branches as well, imho.

So, this isn't as easy as just changing URLs. As far as I can tell, Gitiles only supports one form of "raw" output, and that's as base64-encoded text (https://code.google.com/p/gitiles/issues/detail?id=7).

https://chromium.googlesource.com/chromium/src/net/+/master/http/transport_security_state_static.json?format=TEXT

https://chromium.googlesource.com/chromium/src/net/+/master/http/transport_security_state_static.certs?format=TEXT

We need to have getHSTSPreloadList.js and PreloadedHPKPins.json download those files (as appropriate) and base64 decode them before using them.

Comment 2

[Reed Loden [:reed]]

Note that the 'snionly' option was removed. See https://chromium.googlesource.com/chromium/src/net/+/8761155994b46ee082499c778cd1743fa63ce3bc. I don't think anything we had ever supported it, but would be good to check.

Also, a new compact representation of preloaded domains was added. See https://chromium.googlesource.com/chromium/src/net/+/97f74d049bacc74728d348dcc2f5fafe855e565b. Might be interesting to see about adding something like that for Gecko to replace the current format in nsSTSPreloadList.inc.

Comment 3

[Reed Loden [:reed]]

I have no idea what the right flags are nowadays to request blocking on the appropriate branches. :mmc / :keeler, can one of you take this on? Pretty bad that we didn't notice that our list was out-of-date for two months. :(

Comment 4

[Frederik Braun [:freddy]]

I wonder if it might be possible to have a test that fails when the (seemingly) current URL has not been updated for X weeks.

Comment 5

[[:mmc] Monica Chew (no longer reading bugmail)]

Dana is working on this.

Comment 6

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: patch

This updates the source url and decodes the base64 hosted there.

Comment 7

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8509049 [details] [diff] [review]
patch

Review of attachment 8509049 [details] [diff] [review]:
-----------------------------------------------------------------

Is the auth stopper for the garron bug?

Comment 8

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to [:mmc] Monica Chew (please use needinfo) from comment #7)
> Is the auth stopper for the garron bug?

Actually, that's for another site that appears to request a client certificate. Gecko helpfully tries to open up a dialog so the user can pick one, but since xpcshell has no UI, that doesn't work.

Thanks for the quick review. Unfortunately, I forgot that the HPKP script needs to be updated as well. I'll be uploading a more complete patch shortly.

Comment 9

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: patch v2

Comment 10

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to Dana Keeler (:keeler) [use needinfo?] from comment #8)
> (In reply to [:mmc] Monica Chew (please use needinfo) from comment #7)
> > Is the auth stopper for the garron bug?
> 
> Actually, that's for another site that appears to request a client
> certificate

Or, rather, HTTP auth, I believe.

Comment 11

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8509061 [details] [diff] [review]
patch v2

Review of attachment 8509061 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 12

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Thank you!
https://hg.mozilla.org/integration/mozilla-inbound/rev/e3318daf0ca6

Comment 13

[Frederik Braun [:freddy]]

This seems rather bad to me.
Do you think this warrants requesting an uplift, Dana?

Comment 14

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to Frederik Braun [:freddyb] from comment #13)
> This seems rather bad to me.
> Do you think this warrants requesting an uplift, Dana?

Yes, we should uplift this where appropriate. I want to wait for a successful run of the update script on Saturday, though, before going further.

Comment 15

[Reed Loden [:reed]]

(In reply to Dana Keeler (:keeler) [use needinfo?] from comment #14)
> (In reply to Frederik Braun [:freddyb] from comment #13)
> > This seems rather bad to me.
> > Do you think this warrants requesting an uplift, Dana?
> 
> Yes, we should uplift this where appropriate. I want to wait for a
> successful run of the update script on Saturday, though, before going
> further.

Can we just do a manual run and not wait until Saturday? ... or ask RelEng to run it via the official process but just early?

Comment 16

[[:mmc] Monica Chew (no longer reading bugmail)]

(In reply to Frederik Braun [:freddyb] from comment #13)
> This seems rather bad to me.
> Do you think this warrants requesting an uplift, Dana?

Looking at the last 2 months of changes to https://chromium.googlesource.com/chromium/src/net/+log/97f74d049bacc74728d348dcc2f5fafe855e565b/http/transport_security_state_static.json, the biggest changes we've missed are adding github to HSTS. There was one HSTS removal 7 weeks ago, globalcs.co.uk. That change would have been in Aurora by now. There are no changes in Beta, by design, so that doesn't seem too bad to me.

Coop, would you mind running the update script for HPKP and HSTS manually? If it succeeds, we can ask uplift in the next Aurora build which I think happens on Thursday, then both trees will be in sync by Saturday.

Comment 17

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to [:mmc] Monica Chew (please use needinfo) from comment #16)
> Coop, would you mind running the update script for HPKP and HSTS manually?
> If it succeeds, we can ask uplift in the next Aurora build which I think
> happens on Thursday, then both trees will be in sync by Saturday.

Although, we should probably first either directly land this on mozilla-central or wait for inbound to be merged, since it doesn't look like it has been, yet.

Comment 18

[Reed Loden [:reed]]

(In reply to [:mmc] Monica Chew (please use needinfo) from comment #16)
> (In reply to Frederik Braun [:freddyb] from comment #13)
> > This seems rather bad to me.
> > Do you think this warrants requesting an uplift, Dana?
> 
> Looking at the last 2 months of changes to
> https://chromium.googlesource.com/chromium/src/net/+log/
> 97f74d049bacc74728d348dcc2f5fafe855e565b/http/
> transport_security_state_static.json, the biggest changes we've missed are
> adding github to HSTS. There was one HSTS removal 7 weeks ago,
> globalcs.co.uk. That change would have been in Aurora by now. There are no
> changes in Beta, by design, so that doesn't seem too bad to me.

That link misses a bunch of changes in the last two weeks, including Yahoo domains getting added... Check out https://chromium.googlesource.com/chromium/src/net/+log/master/http/transport_security_state_static.json instead.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/e3318daf0ca6

Comment 20

[Chris Cooper [:coop] (he/him)]

Script has been re-run:

https://hg.mozilla.org/releases/mozilla-aurora/rev/b84b417ad1d5
https://hg.mozilla.org/releases/mozilla-aurora/rev/9014364077a7

https://hg.mozilla.org/mozilla-central/rev/5faccfb61d5c
https://hg.mozilla.org/mozilla-central/rev/7bed20f7b776

Comment 21

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8509061 [details] [diff] [review]
patch v2

Approval Request Comment
[Feature/regressing bug #]: Caused by Chromium moving their source tree to gittiles exclusively. 
[User impact if declined]: We'll be behind updating the HPKP and HSTS preload lists by about 2 months on Aurora.
[Describe test coverage new/current, TBPL]: Update looks good in comment 20.
[Risks and why]: Pretty low. The risk would come from buildbot differing significantly in behavior from coop's run in comment 20.
[String/UUID change made/needed]: None.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/31a5a1aa4b8d

Comment 23

[Reed Loden [:reed]]

(In reply to Chris Cooper [:coop] from comment #20)
> Script has been re-run:
> 
> https://hg.mozilla.org/releases/mozilla-aurora/rev/b84b417ad1d5
> https://hg.mozilla.org/releases/mozilla-aurora/rev/9014364077a7

Since this was run before the fixed script landed on Aurora, do we need to get coop to run it again, or will it get run over the weekend (not sure which branches get the buildbot runs)...

Comment 24

[[:mmc] Monica Chew (no longer reading bugmail)]

Buildbot runs it Saturdays for Nightly and Aurora.