1815090 - Certificate expirations issue that causes multiple failures on all trees

Reporter

Description

•

2 years ago

Push with failures: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&selectedTaskRun=bpTDBp-tTve8IF_qUY2oeg.0&searchStr=windows%2C10%2Cx64%2C2004%2Cwebrender%2Cshippable%2Copt%2Cxpcshell%2Ctests%2Con%2Cmsix%2Cpackages%2Ctest-windows10-64-2004-shippable-qr%2Fopt-xpcshell-msix%2Cx1&revision=40237e7707f1928c84d09cdd2f1fd018a74ea784
Failure log: https://treeherder.mozilla.org/logviewer?job_id=404624596&repo=mozilla-central
Failure log 2: https://treeherder.mozilla.org/logviewer?job_id=404626608&repo=mozilla-esr102

[task 2023-02-05T00:25:53.193Z] 00:25:53     INFO -  TEST-START | uriloader/exthandler/tests/unit/test_punycodeURIs.js
[task 2023-02-05T00:25:53.193Z] 00:25:53     INFO -  TEST-SKIP | uriloader/exthandler/tests/unit/test_punycodeURIs.js | took 0ms
[task 2023-02-05T00:25:53.194Z] 00:25:53     INFO -  Retrying tests that failed when run in parallel.
[task 2023-02-05T00:25:53.195Z] 00:25:53     INFO -  TEST-START | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js
[task 2023-02-05T00:25:53.565Z] 00:25:53  WARNING -  TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js | xpcshell return code: 0
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  TEST-INFO took 370ms
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  >>>>>>>
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2023-02-05T00:25:53.573Z] 00:25:53     INFO -  (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  running event loop
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js | Starting load_good_xml
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  (xpcshell/head.js) | test load_good_xml pending (2)
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2023-02-05T00:25:53.574Z] 00:25:53     INFO -  (xpcshell/head.js) | test run_next_test 1 pending (2)
[task 2023-02-05T00:25:53.575Z] 00:25:53     INFO -  (xpcshell/head.js) | test load_good_xml finished (2)
[task 2023-02-05T00:25:53.575Z] 00:25:53     INFO -  xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js | Starting test_valid_content_signature
[task 2023-02-05T00:25:53.575Z] 00:25:53     INFO -  (xpcshell/head.js) | test test_valid_content_signature pending (2)
[task 2023-02-05T00:25:53.575Z] 00:25:53     INFO -  (xpcshell/head.js) | test run_next_test 1 finished (2)
[task 2023-02-05T00:25:53.576Z] 00:25:53     INFO -  "CONSOLE_MESSAGE: (info) 1675556753532	addons.productaddons	INFO	sending request to: http://localhost:53360/good.xml?validSignature"
[task 2023-02-05T00:25:53.576Z] 00:25:53     INFO -  "CONSOLE_MESSAGE: (info) 1675556753544	addons.productaddons	INFO	Completed downloading document"
[task 2023-02-05T00:25:53.576Z] 00:25:53     INFO -  "CONSOLE_MESSAGE: (error) [JavaScript Error: "XML Parsing Error: syntax error
[task 2023-02-05T00:25:53.576Z] 00:25:53     INFO -  Location: http://localhost:53360/valid_x5u
[task 2023-02-05T00:25:53.577Z] 00:25:53     INFO -  Line Number 1, Column 1:" {file: "http://localhost:53360/valid_x5u" line: 1 column: 1 source: "-----BEGIN CERTIFICATE-----"}]"
[task 2023-02-05T00:25:53.578Z] 00:25:53  WARNING -  TEST-UNEXPECTED-FAIL | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js | test_valid_content_signature - [test_valid_content_signature : 112] Should successfully get addon list, instead failed with Error: Content signature is not valid - false == true
[task 2023-02-05T00:25:53.578Z] 00:25:53     INFO -  Z:/task_167555354424938/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js:test_valid_content_signature:112
[task 2023-02-05T00:25:53.578Z] 00:25:53     INFO -  Z:\task_167555354424938\build\tests\xpcshell\head.js:_do_main:238
[task 2023-02-05T00:25:53.578Z] 00:25:53     INFO -  Z:\task_167555354424938\build\tests\xpcshell\head.js:_execute_test:585
[task 2023-02-05T00:25:53.578Z] 00:25:53     INFO -  -e:null:1
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  exiting test
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  Unexpected exception NS_ERROR_ABORT:
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  _abort_failed_test@Z:\task_167555354424938\build\tests\xpcshell\head.js:863:20
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  do_report_result@Z:\task_167555354424938\build\tests\xpcshell\head.js:964:5
[task 2023-02-05T00:25:53.579Z] 00:25:53     INFO -  Assert<@Z:\task_167555354424938\build\tests\xpcshell\head.js:71:21
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  Assert.prototype.report@resource://testing-common/Assert.sys.mjs:240:10
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  Assert.prototype.ok@resource://testing-common/Assert.sys.mjs:266:10
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  test_valid_content_signature@Z:/task_167555354424938/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_ProductAddonChecker_signatures.js:112:12
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  _do_main@Z:\task_167555354424938\build\tests\xpcshell\head.js:238:6
[task 2023-02-05T00:25:53.580Z] 00:25:53     INFO -  _execute_test@Z:\task_167555354424938\build\tests\xpcshell\head.js:585:5
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  @-e:1:1
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  exiting test
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  "CONSOLE_MESSAGE: (info) 1675556753548	addons.productaddons	WARN	Unexpected invalid content signature found during validation"
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  "Force a GC"
[task 2023-02-05T00:25:53.581Z] 00:25:53     INFO -  PID 4260 | !!! error running onStopped callback: TypeError: callback is not a function
[task 2023-02-05T00:25:53.582Z] 00:25:53     INFO -  <<<<<<<
[task 2023-02-05T00:25:53.582Z] 00:25:53     INFO -  TEST-START | dom/streams/test/xpcshell/large-pipeto.js
[task 2023-02-05T00:25:53.582Z] 00:25:53     INFO -  TEST-SKIP | dom/streams/test/xpcshell/large-pipeto.js | took 1ms
[task 2023-02-05T00:25:53.582Z] 00:25:53     INFO -  TEST-START | netwerk/test/unit_ipc/test_http3_prio_disabled_wrap.js
[task 2023-02-05T00:25:53.583Z] 00:25:53     INFO -  TEST-SKIP | netwerk/test/unit_ipc/test_http3_prio_disabled_wrap.js | took 0ms
[task 2023-02-05T00:25:53.583Z] 00:25:53     INFO -  TEST-START | netwerk/test/unit_ipc/test_http3_prio_enabled_wrap.js
[task 2023-02-05T00:25:53.583Z] 00:25:53     INFO -  TEST-SKIP | netwerk/test/unit_ipc/test_http3_prio_enabled_wrap.js | took 0ms
[task 2023-02-05T00:25:53.584Z] 00:25:53     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_oom_annotation.js
[task 2023-02-05T00:25:53.584Z] 00:25:53     INFO -  TEST-SKIP | toolkit/crashreporter/test/unit/test_oom_annotation.js | took 0ms
[task 2023-02-05T00:25:53.584Z] 00:25:53     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_crashreporter_crash.js
[task 2023-02-05T00:25:53.584Z] 00:25:53     INFO -  TEST-SKIP | toolkit/crashreporter/test/unit/test_crashreporter_crash.js | took 0ms
[task 2023-02-05T00:25:53.585Z] 00:25:53     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_crash_heap_corruption.js
[task 2023-02-05T00:25:53.585Z] 00:25:53     INFO -  TEST-SKIP | toolkit/crashreporter/test/unit/test_crash_heap_corruption.js | took 0ms
[task 2023-02-05T00:25:53.585Z] 00:25:53     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_crash_modules.js
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  TEST-SKIP | toolkit/crashreporter/test/unit/test_crash_modules.js | took 0ms
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  TEST-START | uriloader/exthandler/tests/unit/test_punycodeURIs.js
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  TEST-SKIP | uriloader/exthandler/tests/unit/test_punycodeURIs.js | took 0ms
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  INFO | Result summary:
[task 2023-02-05T00:25:53.586Z] 00:25:53     INFO -  INFO | Passed: 833
[task 2023-02-05T00:25:53.587Z] 00:25:53  WARNING -  INFO | Failed: 1
[task 2023-02-05T00:25:53.587Z] 00:25:53  WARNING -  One or more unittests failed.
[task 2023-02-05T00:25:53.587Z] 00:25:53     INFO -  INFO | Todo: 0
[task 2023-02-05T00:25:53.587Z] 00:25:53     INFO -  INFO | Retried: 9
[task 2023-02-05T00:25:53.587Z] 00:25:53     INFO -  SUITE-END | took 644s
[task 2023-02-05T00:25:53.587Z] 00:25:53     INFO -  Node moz-http2 server shutting down ...
[task 2023-02-05T00:25:53.589Z] 00:25:53     INFO -  http3Server server shutting down ...
[task 2023-02-05T00:25:53.589Z] 00:25:53     INFO -  Http3 server http3Server already dead 3221225506
[task 2023-02-05T00:25:53.632Z] 00:25:53    ERROR - Return code: 1
[task 2023-02-05T00:25:53.632Z] 00:25:53     INFO - TinderboxPrint: xpcshell-xpcshell-msix<br/>833/<em class="testfail">1</em>/0
[task 2023-02-05T00:25:53.633Z] 00:25:53  WARNING - # TBPL FAILURE #
[task 2023-02-05T00:25:53.633Z] 00:25:53  WARNING - setting return code to 2
[task 2023-02-05T00:25:53.633Z] 00:25:53  WARNING - The xpcshell suite: xpcshell-msix ran with return status: FAILURE
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running post-action listener: _package_coverage_data
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running post-action listener: _resource_record_post_action
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running post-action listener: process_java_coverage_data
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - [mozharness: 2023-02-05 00:25:53.633230Z] Finished run-tests step (success)
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - [mozharness: 2023-02-05 00:25:53.633230Z] Running uninstall step.
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running pre-action listener: _resource_record_pre_action
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Running main action method: uninstall
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Getting output from command: ['Z:\\task_167555354424938\\build\\venv\\Scripts\\mozuninstall', 'C:\\Program Files\\WindowsApps\\Mozilla.MozillaFirefoxNightly_111.2302.421.0_x64__gmpnhwe7bv608\\VFS\\ProgramFiles\\MozillaFirefoxNightly Package Root\\firefox.exe']
[task 2023-02-05T00:25:53.633Z] 00:25:53     INFO - Copy/paste: Z:\task_167555354424938\build\venv\Scripts\mozuninstall "C:\Program Files\WindowsApps\Mozilla.MozillaFirefoxNightly_111.2302.421.0_x64__gmpnhwe7bv608\VFS\ProgramFiles\MozillaFirefoxNightly Package Root\firefox.exe"
[task 2023-02-05T00:25:56.520Z] 00:25:56     INFO - Using _rmtree_windows ...
[task 2023-02-05T00:25:56.534Z] 00:25:56     INFO - Running command: del /F /Q "Z:\task_167555354424938\tmpfile_stderr"
[task 2023-02-05T00:25:56.543Z] 00:25:56     INFO - Return code: 0
[task 2023-02-05T00:25:56.543Z] 00:25:56     INFO - Using _rmtree_windows ...
[task 2023-02-05T00:25:56.544Z] 00:25:56     INFO - Running command: del /F /Q "Z:\task_167555354424938\tmpfile_stdout"
[task 2023-02-05T00:25:56.552Z] 00:25:56     INFO - Return code: 0
[task 2023-02-05T00:25:56.552Z] 00:25:56     INFO - Running post-action listener: _resource_record_post_action
[task 2023-02-05T00:25:56.553Z] 00:25:56     INFO - [mozharness: 2023-02-05 00:25:56.552713Z] Finished uninstall step (success)

Ryan VanderMeulen [:RyanVM]

Comment 1

•

2 years ago

Looks like we missed some :(

Flags: needinfo?(jschanck)

Flags: needinfo?(dkeeler)

Ryan VanderMeulen [:RyanVM]

Comment 2

•

2 years ago

This is currently closing all trees.

Severity: -- → S1

tracking-firefox110: --- → ?

tracking-firefox111: --- → ?

Priority: -- → P1

Ryan VanderMeulen [:RyanVM]

Updated

•

2 years ago

status-firefox109: --- → affected

status-firefox110: --- → affected

status-firefox111: --- → affected

status-firefox-esr102: --- → affected

tracking-firefox109: --- → +

tracking-firefox110: ? → +

tracking-firefox111: ? → +

Pulsebot

Comment 3

•

2 years ago

Pushed by archaeopteryx@coole-files.de: https://hg.mozilla.org/mozilla-central/rev/5cc0aa91e9ac temporarily disable test_ProductAddonChecker_signatures.js because test certificates expired. a=me

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Updated

•

2 years ago

Keywords: leave-open

Cristian Tuns

Reporter

Comment 4

•

2 years ago

The patch does not seem to fix the entire issue, we still have a high rate of failures: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=5cc0aa91e9ac91642217adcb0d9c70df2e65ac3c&selectedTaskRun=bIKJkPBsTFu4NjmXPRMD-A.0 not only xpcshells.

Pulsebot

Comment 5

•

2 years ago

Pushed by archaeopteryx@coole-files.de: https://hg.mozilla.org/mozilla-central/rev/726efd2dd1b2 temporarily disable more tests which depend on now expired certificates. a=me https://hg.mozilla.org/mozilla-central/rev/2e6262a410b7 regenerate certificates in build/pgo/certs/. a=me

Iulian Moraru

Comment 6

•

2 years ago

bugherder uplift

https://hg.mozilla.org/releases/mozilla-beta/rev/6bdf7df6d0c9
https://hg.mozilla.org/releases/mozilla-beta/rev/28a7afd22c37
https://hg.mozilla.org/releases/mozilla-beta/rev/25dd50a37c8c

status-firefox110: affected → fixed

Iulian Moraru

Comment 7

•

2 years ago

bugherder uplift

https://hg.mozilla.org/releases/mozilla-release/rev/ea3e57f57719
https://hg.mozilla.org/releases/mozilla-release/rev/3ec28c80bcd6
https://hg.mozilla.org/releases/mozilla-release/rev/48181532b788

status-firefox109: affected → fixed

Iulian Moraru

Comment 8

•

2 years ago

bugherder uplift

https://hg.mozilla.org/releases/mozilla-esr102/rev/c41fcf9a244f
https://hg.mozilla.org/releases/mozilla-esr102/rev/dd09572520ca
https://hg.mozilla.org/releases/mozilla-esr102/rev/609c179ed5c2

status-firefox-esr102: affected → fixed

Comment hidden (Intermittent Failures Robot)

BugBot [:suhaib / :marco/ :calixte]

Comment 10

•

2 years ago

The bug is marked as tracked for firefox111 (nightly). We have limited time to fix this, the soft freeze is in 3 days. However, the bug still isn't assigned.

:ckerschb, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

Flags: needinfo?(ckerschb)

John Schanck [:jschanck]

Assignee

Updated

•

2 years ago

Assignee: nobody → dkeeler

Flags: needinfo?(jschanck)

Flags: needinfo?(ckerschb)

John Schanck [:jschanck]

Assignee

Updated

•

2 years ago

Comment 11

•

2 years ago

Attached file Bug 1815090 - regenerate cose_int_signed_with_pkcs7.zip for test_signed_apps.js. r=keeler — Details

John Schanck [:jschanck]

Assignee

Comment 12

•

2 years ago

Attached file Bug 1815090 - regenerate productaddons test certificates. r=keeler — Details

Depends on D169241

Ryan VanderMeulen [:RyanVM]

Comment 13

•

2 years ago

Can ./mach generate-test-certs also take care of build/pgo/certs/ in the future?

John Schanck [:jschanck]

Assignee

Updated

•

2 years ago

Updated

•

2 years ago

Assignee: dkeeler → jschanck

Flags: needinfo?(dkeeler)

Pulsebot

Comment 14

•

2 years ago

Pushed by jschanck@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/7b052341f8a7 regenerate cose_int_signed_with_pkcs7.zip for test_signed_apps.js. r=keeler https://hg.mozilla.org/integration/autoland/rev/a499d4fa7ff9 regenerate productaddons test certificates. r=keeler

Cristina Horotan [:chorotan]

Comment 15

•

2 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/7b052341f8a7
https://hg.mozilla.org/mozilla-central/rev/a499d4fa7ff9

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Comment 16

•

2 years ago

netwerk/test/unit/test_tls_server.js still needs to get re-enabled.

Dana Keeler (she/her) (use needinfo) [:keeler]

Comment 17

•

2 years ago

Attached file Bug 1815090 - regenerate client-cert.p12 r?jschanck — Details

Pulsebot

Comment 18

•

2 years ago

Pushed by dkeeler@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1c58c31b7ca2 regenerate client-cert.p12 r=jschanck

Marian-Vasile Laza

Comment 19

•

2 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/1c58c31b7ca2

Dana Keeler (she/her) (use needinfo) [:keeler]

Comment 20

•

2 years ago

Comment on attachment 9317952 [details]
Bug 1815090 - regenerate client-cert.p12 r?jschanck

Beta/Release Uplift Approval Request

User impact if declined: Test coverage.
Is this code covered by automated tests?: Yes
Has the fix been verified in Nightly?: Yes
Needs manual test from QE?: No
If yes, steps to reproduce:
List of other uplifts needed: None
Risk to taking this patch: Low
Why is the change risky/not risky? (and alternatives if risky): Test-only changes.
String changes made/needed:
Is Android affected?: Yes

Attachment #9317952 - Flags: approval-mozilla-beta?

Donal Meehan [:dmeehan]

Comment 21

•

2 years ago

Comment on attachment 9317952 [details]
Bug 1815090 - regenerate client-cert.p12 r?jschanck

Approved for 111.0b5

Attachment #9317952 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Donal Meehan [:dmeehan]

Comment 22

•

2 years ago

bugherder uplift

https://hg.mozilla.org/releases/mozilla-beta/rev/0ebd8240f29b

status-firefox111: affected → fixed

Flags: in-testsuite+

Dana Keeler (she/her) (use needinfo) [:keeler]

Updated

•

2 years ago

Status: NEW → RESOLVED

Closed: 2 years ago

Resolution: --- → FIXED

Dana Keeler (she/her) (use needinfo) [:keeler]

Updated

•

2 years ago

Keywords: leave-open

Target Milestone: --- → 111 Branch

Bug 1815090 - regenerate cose_int_signed_with_pkcs7.zip for test_signed_apps.js. r=keeler 2 years ago John Schanck [:jschanck] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1815090 - regenerate productaddons test certificates. r=keeler 2 years ago John Schanck [:jschanck] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1815090 - regenerate client-cert.p12 r?jschanck 2 years ago Dana Keeler (she/her) (use needinfo) [:keeler] 48 bytes, text/x-phabricator-request	dmeehan : approval-mozilla-beta+	Details \| Review