TREES CLOSED - Expired PSM test certificates.
Categories
(Core :: Security: PSM, defect, P1)
Tracking
()
People
(Reporter: malexandru, Assigned: jcj)
References
Details
Attachments
(1 file)
At around 12:30 UTC we started seeing failures regarding expired certificates.
|
||
Comment 1•4 years ago
|
||
At least https://searchfox.org/mozilla-central/source/security/manager/ssl/tests/unit/test_missing_intermediate/missing-intermediate.der has expired.
|
|
||
Comment 2•4 years ago
|
||
Is it possible to determine what certificates need to be updated to fix the issue and be able to reopen trees?
|
||
Updated•4 years ago
|
|
Reporter | |
Comment 3•4 years ago
|
||
Example of failure logs:
- https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287517090&repo=autoland&lineNumber=1618
- https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287519699&repo=autoland&lineNumber=8848
- https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287517657&repo=autoland&lineNumber=1977
- https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=287519864&repo=autoland&lineNumber=2512
|
Assignee | |
Comment 4•4 years ago
|
||
At least https://searchfox.org/mozilla-central/source/security/manager/ssl/tests/unit/test_missing_intermediate/missing-intermediate.der has expired.
In bug 1607845 we regenerated all the .pem files, which is what we use generally across the board. .../missing-intermediate.pem was regenerated, but .../missing-intermediate.der is stale, it wasn't. I'm not even quite sure why it's in-use, will have to dig, and look for other .der files in-tree.
|
|
Assignee | |
Comment 5•4 years ago
|
||
Of course... ha
// NB: missing-intermediate.der won't be regenerated when
// missing-intermediate.pem is. Hopefully by that time we can just use
// missing-intermediate.pem directly.
The only .der files in our tests are:
- modules/libmar/tests/unit/data/mycert.der
- modules/libmar/tests/unit/data/mycert2.der
- modules/libmar/tests/unit/data/mycert3.der
--> all three have 0 seconds of validity - security/manager/ssl/tests/unit/test_missing_intermediate/missing-intermediate.der
--> needs regen - security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.der
--> okay to 2035 - security/manager/ssl/tests/unit/tlsserver/default-ee.der
--> not a real DER file
so I think all that got left behind is xpcshellTestRoot.der.
|
|
Assignee | |
Comment 6•4 years ago
|
||
command:
openssl x509 -in security/manager/ssl/tests/unit/test_missing_intermediate/missing-intermediate.pem -outform der -out security/manager/ssl/tests/unit/test_missing_intermediate/missing-intermediate.der
|
|
Assignee | |
Comment 7•4 years ago
|
||
Other PEMs that expire in 2020:
./security/manager/ssl/tests/unit/test_cert_storage_direct/revoked-cert.pem
Not After : May 21 12:00:00 2020 GMT
--
./security/manager/ssl/tests/unit/test_intermediate_preloads/ee2.pem
Not After : Feb 5 00:00:00 2020 GMT
--
./security/manager/ssl/tests/unit/test_intermediate_preloads/int2.pem
Not After : Feb 5 00:00:00 2020 GMT
--
./security/manager/ssl/tests/unit/test_intermediate_preloads/int.pem
Not After : Feb 5 00:00:00 2020 GMT
--
./security/manager/ssl/tests/unit/test_intermediate_preloads/ee.pem
Not After : Feb 5 00:00:00 2020 GMT
--
./security/manager/ssl/tests/unit/test_cert_sha1/int-pre.pem
Not After : Jan 1 00:00:00 2020 GMT
--
./security/manager/ssl/tests/unit/test_certDB_import/emailEE.pem
Not After : Feb 5 00:00:00 2020 GMT
--
./security/manager/ssl/tests/unit/test_certDB_import/importedCA.pem
Not After : Feb 5 00:00:00 2020 GMT
--
Pretty confident the sha1 one doesn't matter, but I need to rundown what happened to the others, too. These must be the other test failures.
|
|
Assignee | |
Comment 8•4 years ago
|
||
test_cert_storage_direct and test_cert_sha1 should be fine. I have no idea why test_certDB_import and test_intermediate_preloads weren't regenerated with the rest, but I did them by hand. New patch and try push coming.
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 9•4 years ago
|
||
certutil -d ./build/pgo/certs -L -n pgoserver
...
Not After : Wed Feb 05 00:00:00 2020
...
okay, someday we need to figure out how to remember that this file exists and is important
|
|
||
Updated•4 years ago
|
|
|
||
Updated•4 years ago
|
|
||
Comment 10•4 years ago
|
||
Pushed by dluca@mozilla.com: https://hg.mozilla.org/mozilla-central/rev/2319a64a3cd8 Regenerate expired certs not handled by Bug 1607845 a=RyanVM CLOSED TREE
|
||
Comment 11•4 years ago
|
||
| bugherder | ||
|
||
Comment 12•4 years ago
|
||
| bugherder uplift | ||
|
|
||
Comment 13•4 years ago
|
||
| bugherder uplift | ||
|
||
Comment 14•4 years ago
|
||
Update: the patch has been landed on integration and uplifted to release and esr68. Trees have been reopened.
|
|
||
Updated•4 years ago
|
|
||
Comment 15•4 years ago
|
||
Please specify a root cause for this bug. See :tmaity for more information.
|
|
Assignee | |
Comment 16•4 years ago
|
||
Root Cause is that there's A) no automated regeneration of these certificates, we did it manually in bug 1607845 and B) we didn't adequately test future dates to get advance notice of the tree-closing failure, when the manual operation in bug 1607845 missed certs.
Description
•