Closed Bug 1525191 Opened 6 years ago Closed 6 years ago

Cert xpcshell tests are permafailing due to expiration, eg. security/manager/ssl/tests/unit/test_cert_chains.js | xpcshell return code: 0

Categories

(Core :: Security, defect)

Product:
Core
Component:
Security
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla67
Tracking Flags:
Tracking Status
firefox-esr60 65+ fixed
firefox65 --- fixed
firefox66 --- fixed
firefox67 --- fixed

People

(Reporter: dvarga, Assigned: jandem)

References

(Depends on 1 open bug)

Details

Attachments

(3 files, 7 obsolete files)

Bug 1525191 - Regenerate certs
47 bytes, text/x-phabricator-request
Details | Review
Bug 1525191 - Regenerate all .pem.certspec files into their .pems r?ccoroiu
47 bytes, text/x-phabricator-request
Details | Review
Bug 1525191 - ESR backport of cert expiration fixes
47 bytes, text/x-phabricator-request
Details | Review

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=226037795&repo=mozilla-inbound

16:28:56 INFO - TEST-START | security/manager/ssl/tests/unit/test_cert_chains.js
16:28:56 WARNING - TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_cert_chains.js | xpcshell return code: 0
16:28:56 INFO - TEST-INFO took 275ms
16:28:56 INFO - >>>>>>>
16:28:56 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
16:28:56 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
16:28:56 INFO - running event loop
16:28:56 INFO - security/manager/ssl/tests/unit/test_cert_chains.js | Starting
16:28:56 INFO - (xpcshell/head.js) | test pending (2)
16:28:56 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_cert_chains.js | - Binary util BadCertServer should exist - true == true
16:28:56 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_cert_chains.js | - certificate folder (bad_certs) should exist - true == true
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 0 finished (2)
16:28:56 INFO - "CONSOLE_MESSAGE: (info) No chrome package registered for chrome://branding/locale/brand.properties"
16:28:56 INFO - PID 7102 | sending 'GET / HTTP/1.0
16:28:56 INFO - PID 7102 | '
16:28:56 INFO - (xpcshell/head.js) | test pending (2)
16:28:56 INFO - (xpcshell/head.js) | test finished (2)
16:28:56 INFO - PID 7102 | HTTP/1.0 200 OK
16:28:56 INFO - PID 7102 | content-type: text/plain
16:28:56 INFO - PID 7102 | connection: close
16:28:56 INFO - PID 7102 | server: httpd.js
16:28:56 INFO - PID 7102 | date: Tue, 05 Feb 2019 00:28:56 GMT
16:28:56 INFO - PID 7102 | content-length: 3
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 1 pending (2)
16:28:56 INFO - (xpcshell/head.js) | test finished (2)
16:28:56 INFO - security/manager/ssl/tests/unit/test_cert_chains.js | Starting
16:28:56 INFO - (xpcshell/head.js) | test pending (2)

Looks like someone needs to run[1]:

./mach python build/pgo/genpgocert.py

and check in the result but I'm not an expert in this area.

[1] https://searchfox.org/mozilla-central/rev/152993fa346c8fd9296e4cd6622234a664f53341/build/pgo/certs/README#4-5

If it helps, this is what that mach command generated for me on a recent checkout of m-c.

Attachment #9041349 - Attachment description: Bug 1525191 - Regenerated certs → Bug 1525191 - Regenerate certs

That will certainly help. A number of xpcshell test certificates need to be regenerated as well. I was hoping it would be a quick run-the-utility-and-replace-the-certs kind of thing, but it looks like some of the tests need slightly more involved changes (no overlap in validity periods for some fixed-period certificates and their (non-fixed-period) CAs, etc.). I won't have time to do this today, but I can first thing tomorrow morning.

Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/f9b86dec401e
Regenerate certs CLOSED TREE a=tomprince

Scripts:
https://gist.github.com/jcjones/b25e07de3a48c3ed084f0f9e26911693

From the above gist

./jcj-regenerate-certspecs

This is a DER form, not a PEM.

openssl x509 -in security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.pem -outform der > security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.der
rm security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.pem

These don't seem to be checked in

rm services/common/tests/unit/test_blocklist_signatures/*.pem

:keller can you please take a look at Comment 6?

Flags: needinfo?(dkeeler)
Pushed by dluca@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/b6ec07118c70
Regenerate all .pem.certspec files into their .pems r=try a=try CLOSED TREE

Depends on D18663

Depends on D18664

Depends on D18665

Depends on D18667

Pushed by archaeopteryx@coole-files.de:
https://hg.mozilla.org/mozilla-central/rev/66ff28da3e7d
part 1 - Remove bogus lines from pem files. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/df9e185667a3
part 2 - Regenerate zip files in security/manager/ssl/tests/unit/test_signed_apps. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/25e6df21a4ee
part 3 - Update EXPECTED_CHAIN in ssl_error_reports.sjs. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/ad8e75714968
part 4 - Back out some .pem changes from b6ec07118c70. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/4c6fca8134ce
part 5 - Fix test_x509.js for updated certificates. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/aae349bf4115
part 6 - Fix test_content_signing.js for updated certificates. a=bustage-fix
https://hg.mozilla.org/mozilla-central/rev/52b73d447c52
part 7 - Fix test_cert_chains.js for updated certificates. a=bustage-fix

I ended up fixing the test failures this time, but can we please automate this process so we don't have to do this again next year?

Anyway, some notes:

  • Part 2 (test_signed_apps zip files) was necessary to fix test_signed_apps.js

I generated the zip files by uncommenting the code in security/manager/ssl/tests/unit/test_signed_apps/moz.build and then doing a build and copying the files from the obj dir to the source dir.

  • Part 3 (ssl_error_reports.sjs) was necessary to fix browser_ssl_error_reports.js For this I set DEBUG to true in httpd.js and dumped the data we got to figure out the new value for this.

  • Part 4 backed out some .pem changes to fix tests that depended on the old files.

  • Part 5 (test_x509.js). The serial number and dates can be printed like this:

    openssl x509 -in security/manager/ssl/tests/unit/bad_certs/default-ee.pem -text -noout

Maybe someone actually familiar with the code should double check all this.

Assignee: nobody → jdemooij

J.C., can you backport this to ESR, please?

Flags: needinfo?(jjones)

I have reviewed the changes pushed in comment 5, comment 8, and comment 17 and they look correct in sum.

Flags: needinfo?(dkeeler)

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #22)

J.C., can you backport this to ESR, please?

Backport patches are mostly done, patterning off m-c (though the patches don't all apply cleanly). I've still got some failures that I haven't had time to work through: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8031b273b2a105dec1d00bf6e3c80e34c110e9e9

Probably going to have to come back to this tomorrow, I'm afraid.

Attachment #9041410 - Attachment is obsolete: true
Attachment #9041412 - Attachment is obsolete: true
Attachment #9041413 - Attachment is obsolete: true
Attachment #9041414 - Attachment is obsolete: true
Attachment #9041415 - Attachment is obsolete: true
Attachment #9041416 - Attachment is obsolete: true
Attachment #9041417 - Attachment is obsolete: true

(I abandoned the revisions that landed yesterday just to get them out of my "Needs Review" queue.)

Bug 1525191 part 0 - Regenerate pgo certs

Original commit: https://hg.mozilla.org/mozilla-central/rev/f9b86dec401e

Bug 1525191 part 1 - Regenerate all .pem.certspec files into their .pems

ESR backport of these three commits:

https://hg.mozilla.org/mozilla-central/rev/b6ec07118c70
https://hg.mozilla.org/mozilla-central/rev/66ff28da3e7d
https://hg.mozilla.org/mozilla-central/rev/ad8e75714968

... and additionally these tests, which exist in ESR60 but not in 67:

security/manager/ssl/tests/unit/test_ocsp_fetch_method/
security/manager/ssl/tests/unit/test_getchain/

Bug 1525191 part 2 - Regenerate zip files in security/manager/ssl/tests/unit/test_signed_apps

Original commit:
https://hg.mozilla.org/mozilla-central/rev/df9e185667a3

Removed the .zips that don't exist in ESR.

Bug 1525191 part 3 - Update EXPECTED_CHAIN in ssl_error_reports.sjs.

Original commit: https://hg.mozilla.org/mozilla-central/rev/25e6df21a4ee

Bug 1525191 part 4 - Fix test_x509.js for updated certificates.

(Renumbered for ESR)

Original commit: https://hg.mozilla.org/mozilla-central/rev/4c6fca8134ce

Bug 1525191 part 5 - Fix test_content_signing.js for updated certificates.

(Renumbered for ESR)

Original commit: https://hg.mozilla.org/mozilla-central/rev/aae349bf4115

See Also: → 1686615
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: