Cert xpcshell tests are permafailing due to expiration, eg. security/manager/ssl/tests/unit/test_cert_chains.js | xpcshell return code: 0

RESOLVED FIXED in Firefox -esr60

Status

()

defect
--
blocker
RESOLVED FIXED
5 months ago
4 months ago

People

(Reporter: dvarga, Assigned: jandem)

Tracking

(Depends on 1 bug)

unspecified
mozilla67
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr6065+ fixed, firefox65 fixed, firefox66 fixed, firefox67 fixed)

Details

Attachments

(3 attachments, 7 obsolete attachments)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
Reporter

Description

5 months ago

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=226037795&repo=mozilla-inbound

16:28:56 INFO - TEST-START | security/manager/ssl/tests/unit/test_cert_chains.js
16:28:56 WARNING - TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_cert_chains.js | xpcshell return code: 0
16:28:56 INFO - TEST-INFO took 275ms
16:28:56 INFO - >>>>>>>
16:28:56 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
16:28:56 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
16:28:56 INFO - running event loop
16:28:56 INFO - security/manager/ssl/tests/unit/test_cert_chains.js | Starting
16:28:56 INFO - (xpcshell/head.js) | test pending (2)
16:28:56 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_cert_chains.js | - Binary util BadCertServer should exist - true == true
16:28:56 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_cert_chains.js | - certificate folder (bad_certs) should exist - true == true
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 0 finished (2)
16:28:56 INFO - "CONSOLE_MESSAGE: (info) No chrome package registered for chrome://branding/locale/brand.properties"
16:28:56 INFO - PID 7102 | sending 'GET / HTTP/1.0
16:28:56 INFO - PID 7102 | '
16:28:56 INFO - (xpcshell/head.js) | test pending (2)
16:28:56 INFO - (xpcshell/head.js) | test finished (2)
16:28:56 INFO - PID 7102 | HTTP/1.0 200 OK
16:28:56 INFO - PID 7102 | content-type: text/plain
16:28:56 INFO - PID 7102 | connection: close
16:28:56 INFO - PID 7102 | server: httpd.js
16:28:56 INFO - PID 7102 | date: Tue, 05 Feb 2019 00:28:56 GMT
16:28:56 INFO - PID 7102 | content-length: 3
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 1 pending (2)
16:28:56 INFO - (xpcshell/head.js) | test finished (2)
16:28:56 INFO - security/manager/ssl/tests/unit/test_cert_chains.js | Starting
16:28:56 INFO - (xpcshell/head.js) | test pending (2)

Looks like someone needs to run[1]:

./mach python build/pgo/genpgocert.py

and check in the result but I'm not an expert in this area.

[1] https://searchfox.org/mozilla-central/rev/152993fa346c8fd9296e4cd6622234a664f53341/build/pgo/certs/README#4-5

If it helps, this is what that mach command generated for me on a recent checkout of m-c.

Attachment #9041349 - Attachment description: Bug 1525191 - Regenerated certs → Bug 1525191 - Regenerate certs

That will certainly help. A number of xpcshell test certificates need to be regenerated as well. I was hoping it would be a quick run-the-utility-and-replace-the-certs kind of thing, but it looks like some of the tests need slightly more involved changes (no overlap in validity periods for some fixed-period certificates and their (non-fixed-period) CAs, etc.). I won't have time to do this today, but I can first thing tomorrow morning.

Comment 5

5 months ago
Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/f9b86dec401e
Regenerate certs CLOSED TREE a=tomprince

Scripts:
https://gist.github.com/jcjones/b25e07de3a48c3ed084f0f9e26911693

From the above gist

./jcj-regenerate-certspecs

This is a DER form, not a PEM.

openssl x509 -in security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.pem -outform der > security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.der
rm security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.pem

These don't seem to be checked in

rm services/common/tests/unit/test_blocklist_signatures/*.pem

:keller can you please take a look at Comment 6?

Flags: needinfo?(dkeeler)

Comment 8

5 months ago
Pushed by dluca@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/b6ec07118c70
Regenerate all .pem.certspec files into their .pems r=try a=try CLOSED TREE
Assignee

Comment 12

5 months ago

Depends on D18663

Assignee

Comment 13

5 months ago

Depends on D18664

Assignee

Comment 14

5 months ago

Depends on D18665

Assignee

Comment 16

5 months ago

Depends on D18667

Comment 17

5 months ago
Pushed by archaeopteryx@coole-files.de:
https://hg.mozilla.org/mozilla-central/rev/66ff28da3e7d
part 1 - Remove bogus lines from pem files. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/df9e185667a3
part 2 - Regenerate zip files in security/manager/ssl/tests/unit/test_signed_apps. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/25e6df21a4ee
part 3 - Update EXPECTED_CHAIN in ssl_error_reports.sjs. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/ad8e75714968
part 4 - Back out some .pem changes from b6ec07118c70. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/4c6fca8134ce
part 5 - Fix test_x509.js for updated certificates. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/aae349bf4115
part 6 - Fix test_content_signing.js for updated certificates. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/52b73d447c52
part 7 - Fix test_cert_chains.js for updated certificates. a=bustage-fix
Assignee

Comment 19

5 months ago

I ended up fixing the test failures this time, but can we please automate this process so we don't have to do this again next year?

Anyway, some notes:

  • Part 2 (test_signed_apps zip files) was necessary to fix test_signed_apps.js

I generated the zip files by uncommenting the code in security/manager/ssl/tests/unit/test_signed_apps/moz.build and then doing a build and copying the files from the obj dir to the source dir.

  • Part 3 (ssl_error_reports.sjs) was necessary to fix browser_ssl_error_reports.js For this I set DEBUG to true in httpd.js and dumped the data we got to figure out the new value for this.

  • Part 4 backed out some .pem changes to fix tests that depended on the old files.

  • Part 5 (test_x509.js). The serial number and dates can be printed like this:

    openssl x509 -in security/manager/ssl/tests/unit/bad_certs/default-ee.pem -text -noout

Maybe someone actually familiar with the code should double check all this.

Assignee: nobody → jdemooij

J.C., can you backport this to ESR, please?

Flags: needinfo?(jjones)

I have reviewed the changes pushed in comment 5, comment 8, and comment 17 and they look correct in sum.

Flags: needinfo?(dkeeler)

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #22)

J.C., can you backport this to ESR, please?

Backport patches are mostly done, patterning off m-c (though the patches don't all apply cleanly). I've still got some failures that I haven't had time to work through: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8031b273b2a105dec1d00bf6e3c80e34c110e9e9

Probably going to have to come back to this tomorrow, I'm afraid.

Attachment #9041410 - Attachment is obsolete: true
Attachment #9041412 - Attachment is obsolete: true
Attachment #9041413 - Attachment is obsolete: true
Attachment #9041414 - Attachment is obsolete: true
Attachment #9041415 - Attachment is obsolete: true
Attachment #9041416 - Attachment is obsolete: true
Attachment #9041417 - Attachment is obsolete: true
Assignee

Comment 25

5 months ago

(I abandoned the revisions that landed yesterday just to get them out of my "Needs Review" queue.)

Bug 1525191 part 0 - Regenerate pgo certs

Original commit: https://hg.mozilla.org/mozilla-central/rev/f9b86dec401e

Bug 1525191 part 1 - Regenerate all .pem.certspec files into their .pems

ESR backport of these three commits:

https://hg.mozilla.org/mozilla-central/rev/b6ec07118c70
https://hg.mozilla.org/mozilla-central/rev/66ff28da3e7d
https://hg.mozilla.org/mozilla-central/rev/ad8e75714968

... and additionally these tests, which exist in ESR60 but not in 67:

security/manager/ssl/tests/unit/test_ocsp_fetch_method/
security/manager/ssl/tests/unit/test_getchain/

Bug 1525191 part 2 - Regenerate zip files in security/manager/ssl/tests/unit/test_signed_apps

Original commit:
https://hg.mozilla.org/mozilla-central/rev/df9e185667a3

Removed the .zips that don't exist in ESR.

Bug 1525191 part 3 - Update EXPECTED_CHAIN in ssl_error_reports.sjs.

Original commit: https://hg.mozilla.org/mozilla-central/rev/25e6df21a4ee

Bug 1525191 part 4 - Fix test_x509.js for updated certificates.

(Renumbered for ESR)

Original commit: https://hg.mozilla.org/mozilla-central/rev/4c6fca8134ce

Bug 1525191 part 5 - Fix test_content_signing.js for updated certificates.

(Renumbered for ESR)

Original commit: https://hg.mozilla.org/mozilla-central/rev/aae349bf4115

Duplicate of this bug: 1525232
You need to log in before you can comment on or make changes to this bug.