531590 - in-tree certs for ssltunnel+mochitest expire when not regenerated after a year

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The mozilla-central and 1.9.2 trees went very orange for all tests run after 9:45AM PST this morning because the certs in build/pgo/certs/ expired at that time.  I regenerated them in a mozilla-central tree using 'make genservercert' in build/pgo/ and checked in the results to mozilla-central and 1.9.2:
http://hg.mozilla.org/mozilla-central/rev/7b58ad67abe9
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/21547b202482
1.9.1 and other trees may need this as well; I haven't looked yet.

We should:

(1) give these certs a much longer expiration time from generation (currently 120 days, although I'm not sure why this didn't cause problems the last time around)

(2) have a test in the tree that their validity is at least 6 months in the future

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #0)
> (2) have a test in the tree that their validity is at least 6 months in the
> future

To clarify:  the point of doing this is so that we have one test that fails with a clear error message (and instructions on how to fix) well before any other tests start failing with this problem.

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

So the argument to -v is supposed to be months, not days.  One certutil command in the makefile sets certs to expire at 120 months, the other at 12 months.  I'm not sure why the expiration was set when it was given the commit log of those files.

I checked in the UI, and the example.com cert I just generated does, in fact, expire 12 months from now.

It looks like what I just ran didn't regenerate the testing CA cert, but that that one expires in 2018 (10 years after its generation date).


We should have tests checking the expiration of all of these, really...

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I confirmed locally that the tests were also hanging in a 1.9.1 tree, regenerated the cert db again in that tree, and committed:
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/82725e596499

Comment 4

[Honza Bambas (:mayhemer)]

I'll think of this and fix it.

Comment 5

[Honza Bambas (:mayhemer)]

And thanks David for running make genservercert on my behalf.

Comment 6

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

November is here again; do we need to fix this another time?

Comment 7

[(not currently active) Ted Mielczarek]

Can we just make these certs expire in 10 years, and at least push the problem off for a while?

Comment 8

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: make certs expire in 10 years rather than 1

untested

Comment 9

[(no longer active)]

Attachment: Patch (v1)

This increases the default certificate validity period for the self-signed cert that we use for testing to 10 years, and includes new certificates with new validity.

Comment 10

[(no longer active)]

http://tbpl.mozilla.org/?tree=Try&rev=ef142b8407f6

Comment 11

[(no longer active)]

Comment on attachment 572575 [details] [diff] [review]
make certs expire in 10 years rather than 1

This patch is a subset of what I have...

Comment 12

[Honza Bambas (:mayhemer)]

Comment on attachment 572580 [details] [diff] [review]
Patch (v1)

Thanks!

Comment 13

[(no longer active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/82f9cf114090

Comment 14

[Mozilla RelEng Bot]

Try run for ef142b8407f6 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=ef142b8407f6
Results (out of 206 total builds):
    success: 177
    warnings: 21
    other: 8
Builds available at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-ef142b8407f6

Comment 15

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/82f9cf114090

Presume this will be needed on the other trees too...

Comment 16

[(not currently active) Ted Mielczarek]

Comment on attachment 572580 [details] [diff] [review]
Patch (v1)

We need to take this patch on aurora and beta to avoid these tests turning permaorange.

This is a test-only change, it simply updates the expiration date of the SSL certificates we use for Mochitest.

Comment 17

[Alex Keybl [:akeybl]]

Comment on attachment 572580 [details] [diff] [review]
Patch (v1)

[Triage Comment]
Approving test changes for aurora and beta.

Comment 18

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

We still need a bug on regenerating these every 10 years, and probably a test that fails when they're within a year of expiring.

Comment 19

[LegNeato]

http://hg.mozilla.org/releases/mozilla-beta/rev/16a58ed3c639

Looks like it is already on aurora.

Comment 20

[Phil Ringnalda (:philor)]

This component could use a few more approval flags. Gosh darn it, I had to unbust without approval :)

https://hg.mozilla.org/releases/mozilla-1.9.2/rev/5a3c09ad5f4a

Comment 21

[Al Billings [:abillings - ex-MoCo]]

What exactly does QA need to verify here since this is fixed in 1.9.2.25?

Comment 22

[Phil Ringnalda (:philor)]

Not one thing: it's test-only, and I would have been screaming and flapping my arms if it didn't fix the tests when I pushed it.

Comment 23

[Al Billings [:abillings - ex-MoCo]]

Can I get a photo of that?

Thanks, Phil. QA will move on to waiting for 1.9.2.25 official builds then.

Comment 24

[Phil Ringnalda (:philor)]

(In reply to David Baron :dbaron: ⌚️UTC-8 from comment #2)
> It looks like what I just ran didn't regenerate the testing CA cert, but
> that that one expires in 2018 (10 years after its generation date).

Remember when "expires in 2018" seemed like a long time in the future, rather than like 103 days from now?

Comment 25

[Phil Ringnalda (:philor)]

Or maybe like today, not sure whether you meant the one that expired this afternoon and is causing bug 1435644, or /build/pgo/certs/pgoca.ca which expires May 18th.