Closed Bug 1815090 Opened 1 year ago Closed 1 year ago

Certificate expirations issue that causes multiple failures on all trees

Categories

(Core :: Security: PSM, defect, P1)

Product:
Core
Component:
Security: PSM
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
111 Branch
Tracking Flags:
Tracking Status
firefox-esr102 --- fixed
firefox109 + fixed
firefox110 + fixed
firefox111 + fixed

People

(Reporter: ctuns, Assigned: jschanck)

References

Details

Attachments

(3 files)

Bug 1815090 - regenerate cose_int_signed_with_pkcs7.zip for test_signed_apps.js. r=keeler
48 bytes, text/x-phabricator-request
Details | Review
Bug 1815090 - regenerate productaddons test certificates. r=keeler
48 bytes, text/x-phabricator-request
Details | Review
Bug 1815090 - regenerate client-cert.p12 r?jschanck
48 bytes, text/x-phabricator-request
dmeehan
: approval-mozilla-beta+
Details | Review

Push with failures: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&selectedTaskRun=bpTDBp-tTve8IF_qUY2oeg.0&searchStr=windows%2C10%2Cx64%2C2004%2Cwebrender%2Cshippable%2Copt%2Cxpcshell%2Ctests%2Con%2Cmsix%2Cpackages%2Ctest-windows10-64-2004-shippable-qr%2Fopt-xpcshell-msix%2Cx1&revision=40237e7707f1928c84d09cdd2f1fd018a74ea784
Failure log: https://treeherder.mozilla.org/logviewer?job_id=404624596&repo=mozilla-central
Failure log 2: https://treeherder.mozilla.org/logviewer?job_id=404626608&repo=mozilla-esr102

[task 2023-02-05T00:25:53.193Z] 00:25:53     INFO -  TEST-START | uriloader/exthandler/tests/unit/test_punycodeURIs.js
[task 2023-02-05T00:25:53.193Z] 00:25:53     INFO -  TEST-SKIP | uriloader/exthandler/tests/unit/test_punycodeURIs.js | took 0ms
[task 2023-02-05T00:25:53.194Z] 00:25:53     INFO -  Retrying tests that failed when run in parallel.
[task 2023-02-05T00:25:53.195Z] 00:25:53     INFO -  TEST-START | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js
[task 2023-02-05T00:25:53.565Z] 00:25:53  WARNING -  TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js | xpcshell return code: 0
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  TEST-INFO took 370ms
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  >>>>>>>
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  running event loop
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js | Starting load_good_xml
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  (xpcshell/head.js) | test load_good_xml pending (2)
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  (xpcshell/head.js) | test run_next_test 1 pending (2)
[task 2023-02-05T00:25:53.575Z] 00:25:53     INFO -  (xpcshell/head.js) | test load_good_xml finished (2)
[task 2023-02-05T00:25:53.575Z] 00:25:53     INFO -  xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js | Starting test_valid_content_signature
[task 2023-02-05T00:25:53.575Z] 00:25:53     INFO -  (xpcshell/head.js) | test test_valid_content_signature pending (2)
[task 2023-02-05T00:25:53.575Z] 00:25:53     INFO -  (xpcshell/head.js) | test run_next_test 1 finished (2)
[task 2023-02-05T00:25:53.576Z] 00:25:53     INFO -  "CONSOLE_MESSAGE: (info) 1675556753532	addons.productaddons	INFO	sending request to: http://localhost:53360/good.xml?validSignature"
[task 2023-02-05T00:25:53.576Z] 00:25:53     INFO -  "CONSOLE_MESSAGE: (info) 1675556753544	addons.productaddons	INFO	Completed downloading document"
[task 2023-02-05T00:25:53.576Z] 00:25:53     INFO -  "CONSOLE_MESSAGE: (error) [JavaScript Error: "XML Parsing Error: syntax error
[task 2023-02-05T00:25:53.576Z] 00:25:53     INFO -  Location: http://localhost:53360/valid_x5u
[task 2023-02-05T00:25:53.577Z] 00:25:53     INFO -  Line Number 1, Column 1:" {file: "http://localhost:53360/valid_x5u" line: 1 column: 1 source: "-----BEGIN CERTIFICATE-----"}]"
[task 2023-02-05T00:25:53.578Z] 00:25:53  WARNING -  TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js | test_valid_content_signature - [test_valid_content_signature : 112] Should successfully get addon list, instead failed with Error: Content signature is not valid - false == true
[task 2023-02-05T00:25:53.578Z] 00:25:53     INFO -  Z:/task_167555354424938/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js:test_valid_content_signature:112
[task 2023-02-05T00:25:53.578Z] 00:25:53     INFO -  Z:\task_167555354424938\build\tests\xpcshell\head.js:_do_main:238
[task 2023-02-05T00:25:53.578Z] 00:25:53     INFO -  Z:\task_167555354424938\build\tests\xpcshell\head.js:_execute_test:585
[task 2023-02-05T00:25:53.578Z] 00:25:53     INFO -  -e:null:1
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  exiting test
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  Unexpected exception NS_ERROR_ABORT:
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  _abort_failed_test@Z:\task_167555354424938\build\tests\xpcshell\head.js:863:20
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  do_report_result@Z:\task_167555354424938\build\tests\xpcshell\head.js:964:5
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  Assert<@Z:\task_167555354424938\build\tests\xpcshell\head.js:71:21
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  Assert.prototype.report@resource://testing-common/Assert.sys.mjs:240:10
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  Assert.prototype.ok@resource://testing-common/Assert.sys.mjs:266:10
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  test_valid_content_signature@Z:/task_167555354424938/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js:112:12
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  _do_main@Z:\task_167555354424938\build\tests\xpcshell\head.js:238:6
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  _execute_test@Z:\task_167555354424938\build\tests\xpcshell\head.js:585:5
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  @-e:1:1
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  exiting test
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  "CONSOLE_MESSAGE: (info) 1675556753548	addons.productaddons	WARN	Unexpected invalid content signature found during validation"
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  "Force a GC"
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  PID 4260 | !!! error running onStopped callback: TypeError: callback is not a function
[task 2023-02-05T00:25:53.582Z] 00:25:53     INFO -  <<<<<<<
[task 2023-02-05T00:25:53.582Z] 00:25:53     INFO -  TEST-START | dom/streams/test/xpcshell/large-pipeto.js
[task 2023-02-05T00:25:53.582Z] 00:25:53     INFO -  TEST-SKIP | dom/streams/test/xpcshell/large-pipeto.js | took 1ms
[task 2023-02-05T00:25:53.582Z] 00:25:53     INFO -  TEST-START | netwerk/test/unit_ipc/test_http3_prio_disabled_wrap.js
[task 2023-02-05T00:25:53.583Z] 00:25:53     INFO -  TEST-SKIP | netwerk/test/unit_ipc/test_http3_prio_disabled_wrap.js | took 0ms
[task 2023-02-05T00:25:53.583Z] 00:25:53     INFO -  TEST-START | netwerk/test/unit_ipc/test_http3_prio_enabled_wrap.js
[task 2023-02-05T00:25:53.583Z] 00:25:53     INFO -  TEST-SKIP | netwerk/test/unit_ipc/test_http3_prio_enabled_wrap.js | took 0ms
[task 2023-02-05T00:25:53.584Z] 00:25:53     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_oom_annotation.js
[task 2023-02-05T00:25:53.584Z] 00:25:53     INFO -  TEST-SKIP | toolkit/crashreporter/test/unit/test_oom_annotation.js | took 0ms
[task 2023-02-05T00:25:53.584Z] 00:25:53     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_crashreporter_crash.js
[task 2023-02-05T00:25:53.584Z] 00:25:53     INFO -  TEST-SKIP | toolkit/crashreporter/test/unit/test_crashreporter_crash.js | took 0ms
[task 2023-02-05T00:25:53.585Z] 00:25:53     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_crash_heap_corruption.js
[task 2023-02-05T00:25:53.585Z] 00:25:53     INFO -  TEST-SKIP | toolkit/crashreporter/test/unit/test_crash_heap_corruption.js | took 0ms
[task 2023-02-05T00:25:53.585Z] 00:25:53     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_crash_modules.js
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  TEST-SKIP | toolkit/crashreporter/test/unit/test_crash_modules.js | took 0ms
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  TEST-START | uriloader/exthandler/tests/unit/test_punycodeURIs.js
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  TEST-SKIP | uriloader/exthandler/tests/unit/test_punycodeURIs.js | took 0ms
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  INFO | Result summary:
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  INFO | Passed: 833
[task 2023-02-05T00:25:53.587Z] 00:25:53  WARNING -  INFO | Failed: 1
[task 2023-02-05T00:25:53.587Z] 00:25:53  WARNING -  One or more unittests failed.
[task 2023-02-05T00:25:53.587Z] 00:25:53     INFO -  INFO | Todo: 0
[task 2023-02-05T00:25:53.587Z] 00:25:53     INFO -  INFO | Retried: 9
[task 2023-02-05T00:25:53.587Z] 00:25:53     INFO -  SUITE-END | took 644s
[task 2023-02-05T00:25:53.587Z] 00:25:53     INFO -  Node moz-http2 server shutting down ...
[task 2023-02-05T00:25:53.589Z] 00:25:53     INFO -  http3Server server shutting down ...
[task 2023-02-05T00:25:53.589Z] 00:25:53     INFO -  Http3 server http3Server already dead 3221225506
[task 2023-02-05T00:25:53.632Z] 00:25:53    ERROR - Return code: 1
[task 2023-02-05T00:25:53.632Z] 00:25:53     INFO - TinderboxPrint: xpcshell-xpcshell-msix<br/>833/<em class="testfail">1</em>/0
[task 2023-02-05T00:25:53.633Z] 00:25:53  WARNING - # TBPL FAILURE #
[task 2023-02-05T00:25:53.633Z] 00:25:53  WARNING - setting return code to 2
[task 2023-02-05T00:25:53.633Z] 00:25:53  WARNING - The xpcshell suite: xpcshell-msix ran with return status: FAILURE
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running post-action listener: _package_coverage_data
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running post-action listener: _resource_record_post_action
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running post-action listener: process_java_coverage_data
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - [mozharness: 2023-02-05 00:25:53.633230Z] Finished run-tests step (success)
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - [mozharness: 2023-02-05 00:25:53.633230Z] Running uninstall step.
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running pre-action listener: _resource_record_pre_action
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running main action method: uninstall
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Getting output from command: ['Z:\\task_167555354424938\\build\\venv\\Scripts\\mozuninstall', 'C:\\Program Files\\WindowsApps\\Mozilla.MozillaFirefoxNightly_111.2302.421.0_x64__gmpnhwe7bv608\\VFS\\ProgramFiles\\MozillaFirefoxNightly Package Root\\firefox.exe']
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Copy/paste: Z:\task_167555354424938\build\venv\Scripts\mozuninstall "C:\Program Files\WindowsApps\Mozilla.MozillaFirefoxNightly_111.2302.421.0_x64__gmpnhwe7bv608\VFS\ProgramFiles\MozillaFirefoxNightly Package Root\firefox.exe"
[task 2023-02-05T00:25:56.520Z] 00:25:56     INFO - Using _rmtree_windows ...
[task 2023-02-05T00:25:56.534Z] 00:25:56     INFO - Running command: del /F /Q "Z:\task_167555354424938\tmpfile_stderr"
[task 2023-02-05T00:25:56.543Z] 00:25:56     INFO - Return code: 0
[task 2023-02-05T00:25:56.543Z] 00:25:56     INFO - Using _rmtree_windows ...
[task 2023-02-05T00:25:56.544Z] 00:25:56     INFO - Running command: del /F /Q "Z:\task_167555354424938\tmpfile_stdout"
[task 2023-02-05T00:25:56.552Z] 00:25:56     INFO - Return code: 0
[task 2023-02-05T00:25:56.552Z] 00:25:56     INFO - Running post-action listener: _resource_record_post_action
[task 2023-02-05T00:25:56.553Z] 00:25:56     INFO - [mozharness: 2023-02-05 00:25:56.552713Z] Finished uninstall step (success)

Looks like we missed some :(

Flags: needinfo?(jschanck)
Flags: needinfo?(dkeeler)

This is currently closing all trees.

Severity: -- → S1
tracking-firefox110: --- → ?
tracking-firefox111: --- → ?
Priority: -- → P1
Pushed by archaeopteryx@coole-files.de:
https://hg.mozilla.org/mozilla-central/rev/5cc0aa91e9ac
temporarily disable test_ProductAddonChecker_signatures.js because test certificates expired. a=me
Pushed by archaeopteryx@coole-files.de:
https://hg.mozilla.org/mozilla-central/rev/726efd2dd1b2
temporarily disable more tests which depend on now expired certificates. a=me
https://hg.mozilla.org/mozilla-central/rev/2e6262a410b7
regenerate certificates in build/pgo/certs/. a=me

The bug is marked as tracked for firefox111 (nightly). We have limited time to fix this, the soft freeze is in 3 days. However, the bug still isn't assigned.

:ckerschb, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

Flags: needinfo?(ckerschb)
Assignee: nobody → dkeeler
Flags: needinfo?(jschanck)
Flags: needinfo?(ckerschb)
See Also: → 1815768

Can ./mach generate-test-certs also take care of build/pgo/certs/ in the future?

See Also: → 1815770
Assignee: dkeeler → jschanck
Flags: needinfo?(dkeeler)
Pushed by jschanck@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7b052341f8a7
regenerate cose_int_signed_with_pkcs7.zip for test_signed_apps.js. r=keeler
https://hg.mozilla.org/integration/autoland/rev/a499d4fa7ff9
regenerate productaddons test certificates. r=keeler

netwerk/test/unit/test_tls_server.js still needs to get re-enabled.

Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1c58c31b7ca2
regenerate client-cert.p12 r=jschanck

Comment on attachment 9317952 [details]
Bug 1815090 - regenerate client-cert.p12 r?jschanck

Beta/Release Uplift Approval Request

  • User impact if declined: Test coverage.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Test-only changes.
  • String changes made/needed:
  • Is Android affected?: Yes
Attachment #9317952 - Flags: approval-mozilla-beta?

Comment on attachment 9317952 [details]
Bug 1815090 - regenerate client-cert.p12 r?jschanck

Approved for 111.0b5

Attachment #9317952 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Keywords: leave-open
Target Milestone: --- → 111 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: