1435946 - /build/pgo/certs/pgoca.ca expires 2018-05-18

Status: VERIFIED FIXED

(Firefox Build System :: General, defect, P1)

Description

[Phil Ringnalda (:philor)]

Conveniently, 60 will be on release before then, so we don't have to mess with 59 or 58, but inconveniently esr52 doesn't die until August, so we need to do it. I thought this might be simple enough I could just do it, but since just running --gen-ca both produces maybe-nothing errors and produces busted tests, and running --gen-server afterward in case that's actually required results in creating new cert9.db/key4.db rather than just modifying the old cert8.db/key3.db, I'll leave it to someone who has some idea what they are doing.

Or, we could do what glandium thinks should be done, just generate it at build-time instead of trying to remember to do it again in 2028.

Comment 1

[(not currently active) Ted Mielczarek]

(In reply to Phil Ringnalda (:philor) from comment #0)
> Conveniently, 60 will be on release before then, so we don't have to mess
> with 59 or 58, but inconveniently esr52 doesn't die until August, so we need
> to do it. I thought this might be simple enough I could just do it, but
> since just running --gen-ca both produces maybe-nothing errors and produces
> busted tests, and running --gen-server afterward in case that's actually
> required results in creating new cert9.db/key4.db rather than just modifying
> the old cert8.db/key3.db, I'll leave it to someone who has some idea what
> they are doing.

This is OK, NSS just occasionally bumps the version number of their datafiles when the format changes. We should fix the README to better indicate that, and make sure the test harness scripts that copy those files handle version number changes appropriately.

> Or, we could do what glandium thinks should be done, just generate it at
> build-time instead of trying to remember to do it again in 2028.

We actually did generate them at build-time originally, but we changed it in bug 459699 because that doesn't work in cross-compiles, where we don't have host NSS tools.

Comment 2

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=55b3d95260a43df25c8f9b50e8e6834c084e66e7

Comment 3

[Mike Hommey [:glandium]]

<glandium> ted: the code that uses the ca file uses certutil right after that
<glandium> also, it's for pgo, pgo doesn't work on cross-builds anyways

Comment 4

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=55b3d95260a43df25c8f9b50e8e6834c084e66e7

Those mochitests are very orange. From one of the logs:
[task 2018-02-06T16:40:59.076Z] 16:40:59     INFO -  pk12util: PKCS12 decode import bags failed: SEC_ERROR_REUSED_ISSUER_AND_SERIAL: You are attempting to import a cert with the same issuer/serial as an existing cert, but that is not the same cert.

...I'm guessing I screwed something up here.

Comment 5

[Phil Ringnalda (:philor)]

Actually, the way relman shoves release dates around, only having 10 days after when 59 is currently scheduled to be EOL probably isn't enough and we need to update it too just in case it has to live past May 18th.

Comment 6

[Phil Ringnalda (:philor)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ae09c17d8a205667b88bc293191778670a472896 is considerably more successful, from just manually removing cert8.db and key3.db before generating new ones, so I think maybe unlinkDbFiles isn't working (though I removed them before running --gen-ca, so maybe it works but needs to be done by --gen-ca, not just by --gen-server?).

At least some of the remaining orange is SEC_ERROR_REUSED_ISSUER_AND_SERIAL, so without knowing what any part of the script is actually doing, I wonder if that means https://hg.mozilla.org/try/rev/69a7e1c1b89333b1dadd1548741d10905472fadc#l1.258 was actually required (despite it looking to the naive reader like what we used to do was iterate through the set of locations, adding one, then removing that one we had just added and adding the next, then removing the second and adding the third).

Comment 7

[(not currently active) Ted Mielczarek]

(In reply to Phil Ringnalda (:philor) from comment #6)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=ae09c17d8a205667b88bc293191778670a472896 is
> considerably more successful, from just manually removing cert8.db and
> key3.db before generating new ones, so I think maybe unlinkDbFiles isn't
> working (though I removed them before running --gen-ca, so maybe it works
> but needs to be done by --gen-ca, not just by --gen-server?).

In my patches I had to hack some things to actually work. `--gen-ca` doesn't create those files, `--gen-server` does.


> At least some of the remaining orange is SEC_ERROR_REUSED_ISSUER_AND_SERIAL,
> so without knowing what any part of the script is actually doing, I wonder
> if that means
> https://hg.mozilla.org/try/rev/69a7e1c1b89333b1dadd1548741d10905472fadc#l1.
> 258 was actually required (despite it looking to the naive reader like what
> we used to do was iterate through the set of locations, adding one, then
> removing that one we had just added and adding the next, then removing the
> second and adding the third).

Yeah, I think maybe I removed that in error? I'm not entirely sure.

Comment 8

[Phil Ringnalda (:philor)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d0512d3529e7974e80db993ffecabe92ce82c7cc was some thrashing around (the mq model, where you hide your mistakes, suited me much better) to put the delete line back in, which does result in a bit of green, but still with the SEC_ERROR_REUSED_ISSUER_AND_SERIAL. Because I'm curious like that, https://treeherder.mozilla.org/#/jobs?repo=try&revision=19161893460f480c07ca35f5b395ee86321ad727 is the result of just running --gen-server on m-c tip with an unaltered genpgocert.py, just to make sure we weren't seeing any bustage from people adding things that would be broken by any run of --gen-server, but just adding a new cert9.db/key4.db and removing cert8.db/key3.db doesn't cause any of the bustage.

People may very well have added thing which depend on pgoca.ca but don't regenerate from --gen-server, though. I tried to follow what https://hg.mozilla.org/mozilla-central/annotate/0ac953fcddf10132eaecdb753d72b2ba5a43c32a/devtools/client/webconsole/test/browser_console_certificate_imminent_distrust.js#l6 was saying, but failed completely.

Comment 9

[Phil Ringnalda (:philor)]

There's no question this blocks a 60.0.1, and 61, and 62, and esr60, but I'd argue that it also blocks 60, because it would be a jerk move on our part to tag a 60.0 revision which could only be built or have tests run against it for 9 days after it was released.

Comment 10

[Julien Cristau [:jcristau]]

Yup, makes sense.  Tracking as blocker.

Comment 11

[Julien Cristau [:jcristau]]

Mike, Ted, can one of you pick this up, or help find an owner?

Comment 12

[Mike Hommey [:glandium]]

So, the README specifically says --gen-server doesn't generate all the certificates, and points to bug 1441338.
Some of the certificates in there appear to be the same as pgoca.ca (so, expire on the same date), presumably with some flags set:
- pgo temporary ca
- escapeattack1
- dynamicPinningGood
- sha1_end_entity
- escapeattack2
- expired
- sha256_end_entity
- pgo server certificate
- imminently_distrusted

The other certificates have different expiry times:
- selfsigned, expires on Jan 21 09:42:18 2019 GMT
- Unknown CA, expires on Jan 21 09:42:55 2019 GMT
- untrustedandexpired, expires on Jan 21 09:42:55 2019 GMT
- alternateTrustedAuthority, expires on Sep 25 21:21:54 2024 GMT
- staticPinningBad, expires on Sep 25 21:21:54 2024 GMT
- bug413909cert, expires on May 26 20:40:36 2018 GMT
- untrusted, expires on Jan 21 09:42:55 2019 GMT
- dynamicPinningBad, expires on Sep 25 21:21:54 2024 GMT

So one expires slightly after pgoca.ca, and 4 others expire next year, well within esr60 lifetime.

Coincidentally, the certificate database has been last changed *after* this bug was filed, and we now have cert9.db/key4.db instead of cert8.db/key3.db. The comment about the scripts not dealing with all the certificates and pointing to bug 1441338 was added then, in bug 1439378.

So I'll just punt to jcj, who made those changes, since he apparently knows more about this than I do.

Comment 13

[J.C. Jones [:jcj] (he/him)]

OK, I'll get started on this on Tuesday, via Bug 1441338 (I'm on PTO Thurs/Fri). The changes that come from it will be upliftable without too much concern.

Comment 14

[Julien Cristau [:jcristau]]

Fixed in bug 1441338

Comment 15

[J.C. Jones [:jcj] (he/him)]

Just an update: the patch for ESR52 is attached in bug 1441338. There are some buildbot issues testing it on Windows platforms though, but as a cert DB update it should not be affected by platform.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

We can call this bug verified when we're still successfully building on the 19th :)

Comment 17

[Ryan VanderMeulen [:RyanVM]]

It's after May 18 and PGO builds are still working :).

Comment 18

[J.C. Jones [:jcj] (he/him)]

\o/

thank goodness...