1989205 - Permafail across trees TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js | Test timed out -

Reporter

Description

•

6 months ago

treeherder

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=527550262&repo=autoland&task=fa9HMji6TPKeCXuMQK0KEw.0
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fa9HMji6TPKeCXuMQK0KEw/runs/0/artifacts/public/logs/live_backing.log

[task 2025-09-18T05:25:43.132+00:00] 05:25:43     INFO - TEST-PASS | toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js | Should be no active installs - 
[task 2025-09-18T05:25:43.133+00:00] 05:25:43     INFO - Waiting for addon-webext-permissions notification
[task 2025-09-18T05:25:43.134+00:00] 05:25:43     INFO - Observed addon-install-started for 1 installs
[task 2025-09-18T05:25:43.134+00:00] 05:25:43     INFO - Install of http://example.com/browser/toolkit/mozapps/extensions/test/xpinstall/recommended.xpi was in state 0
[task 2025-09-18T05:25:43.135+00:00] 05:25:43     INFO - Console message: 1758172960115	addons.manager	INFO	Addon download before validation.
[task 2025-09-18T05:25:43.136+00:00] 05:25:43     INFO - Console message: 1758172960132	addons.xpi	DEBUG	Download started for http://example.com/browser/toolkit/mozapps/extensions/test/xpinstall/recommended.xpi to file C:\Users\task_175817143829905\AppData\Local\Temp\tmp-dup.xpi
[task 2025-09-18T05:25:43.136+00:00] 05:25:43     INFO - Console message: 1758172960133	addons.xpi	DEBUG	Download of http://example.com/browser/toolkit/mozapps/extensions/test/xpinstall/recommended.xpi completed.
[task 2025-09-18T05:25:43.136+00:00] 05:25:43     INFO - Observed addon-install-blocked for 1 installs
[task 2025-09-18T05:25:43.137+00:00] 05:25:43     INFO - Install of http://example.com/browser/toolkit/mozapps/extensions/test/xpinstall/recommended.xpi was in state 3
[task 2025-09-18T05:25:43.137+00:00] 05:25:43     INFO - Console message: 1758172960144	addons.repository	DEBUG	cacheAddons: enabled false IDs ["{811d77f1-f306-4187-9251-b4ff99bad60b}"]
[task 2025-09-18T05:25:43.138+00:00] 05:25:43     INFO - Buffered messages logged at 05:22:55
[task 2025-09-18T05:25:43.138+00:00] 05:25:43     INFO - Console message: 1758172975323	addons.manager	DEBUG	Background update check beginning
[task 2025-09-18T05:25:43.139+00:00] 05:25:43     INFO - Console message: 1758172975323	addons.manager	DEBUG	Background update check complete
[task 2025-09-18T05:25:43.139+00:00] 05:25:43     INFO - Buffered messages logged at 05:23:27
[task 2025-09-18T05:25:43.139+00:00] 05:25:43     INFO - Longer timeout required, waiting longer...  Remaining timeouts: 3
[task 2025-09-18T05:25:43.140+00:00] 05:25:43     INFO - Buffered messages logged at 05:24:12
[task 2025-09-18T05:25:43.140+00:00] 05:25:43     INFO - Longer timeout required, waiting longer...  Remaining timeouts: 2
[task 2025-09-18T05:25:43.141+00:00] 05:25:43     INFO - Buffered messages logged at 05:24:57
[task 2025-09-18T05:25:43.141+00:00] 05:25:43     INFO - Longer timeout required, waiting longer...  Remaining timeouts: 1
[task 2025-09-18T05:25:43.142+00:00] 05:25:43     INFO - Buffered messages finished
[task 2025-09-18T05:25:43.142+00:00] 05:25:43     INFO - TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js | Test timed out - 
[task 2025-09-18T05:25:43.142+00:00] 05:25:43     INFO - GECKO(2628) | Completed ShutdownLeaks collections in process 2096
[task 2025-09-18T05:25:43.143+00:00] 05:25:43     INFO - TEST-START | Shutdown
[task 2025-09-18T05:25:43.143+00:00] 05:25:43     INFO - Browser Chrome Test Summary
[task 2025-09-18T05:25:43.144+00:00] 05:25:43     INFO - Passed:  126
[task 2025-09-18T05:25:43.144+00:00] 05:25:43     INFO - Failed:  1
[task 2025-09-18T05:25:43.145+00:00] 05:25:43     INFO - Todo:    0
[task 2025-09-18T05:25:43.145+00:00] 05:25:43     INFO - Mode:    e10s
[task 2025-09-18T05:25:43.146+00:00] 05:25:43     INFO - *** End BrowserChrome Test Results ***
[task 2025-09-18T05:25:43.146+00:00] 05:25:43     INFO - GECKO(2628) | Exiting due to channel error.
[task 2025-09-18T05:25:43.146+00:00] 05:25:43     INFO - GECKO(2628) | Exiting due to channel error.
[task 2025-09-18T05:25:43.147+00:00] 05:25:43     INFO - TEST-INFO | Main app process: exit 0
[task 2025-09-18T05:25:43.147+00:00] 05:25:43     INFO - TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js | Application shut down (without crashing) in the middle of a test!
[task 2025-09-18T05:25:43.147+00:00] 05:25:43     INFO - TEST-INFO took 184500ms
[task 2025-09-18T05:25:43.148+00:00] 05:25:43     INFO - runtests.py | Application ran for: 0:03:18.345741

Serban Stanca [:SerbanS]

Comment 1

•

6 months ago

This is permafailing on all platforms except on Android. Couldn't find something relevant about this even after some retriggers and backfills.

Hi Rob! Could you maybe help us finding a possible cause for this one?

Thank you!

Severity: S4 → --

Flags: needinfo?(rob)

Priority: P5 → --

Serban Stanca [:SerbanS]

Comment 2

•

6 months ago

It also fails on mozilla-beta: https://treeherder.mozilla.org/jobs?repo=mozilla-beta&group_state=expanded&resultStatus=pending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&searchStr=linux%2C24.04%2Cdebug%2Cmochitests%2Cwith%2Csoftware%2Cwebrender%2Cenabled%2Ctest-linux2404-64%2Fdebug-mochitest-browser-chrome-swr%2Cbc11&revision=1db26b1f0b8a6351bcb359f30fc2a6af8dfdd3b8&selectedTaskRun=BhGmRwFMRy-wcb8gJF-MpA.0.

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Comment 3

•

6 months ago

Could a person from your team investigate this permanent failure? It affects at least trunk, beta and release. The recommended.xpi file installs fine locally and the screenshot of such a failing task shows it gets stuck earlier at the install prompt.

status-firefox143: --- → affected

status-firefox144: --- → affected

status-firefox145: --- → affected

Flags: needinfo?(mixedpuppy)

Summary: Perma TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js | Test timed out - → Permafail across trees TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js | Test timed out -

Rob Wu [:robwu]

Comment 4

•

6 months ago

The failure is due to a time bomb. Recommendations are currently only valid for 5 years, and the test file introduced in bug 1659530 is now past that age.

recommended.xpi is no longer recommended: https://robwu.nl/crxviewer/?crx=https%3A%2F%2Fsearchfox.org%2Ffirefox-main%2Frev%2Fe4d8867f696477f5ec96db70d584716c6edb1213%2Ftoolkit%2Fmozapps%2Fextensions%2Ftest%2Fxpinstall%2Frecommended.xpi&q=!not_after&qf=mozilla-recommendation.json&qb=1&qh=0&qi=1

{
    "addon_id": "{811d77f1-f306-4187-9251-b4ff99bad60b}",
    "states": [
        "line"
    ],
    "validity": {
        "not_after": "2025-09-17T22:50:45Z",
        "not_before": "2020-09-17T16:50:45Z"
    },
    "schema_version": 1
}

If we haven't done so, we should consider switching to the AMO stage cert (context: bug 1886252).

Flags: needinfo?(rob)

Flags: needinfo?(mixedpuppy)

Updated

•

6 months ago

See Also: → https://mozilla-hub.atlassian.net/browse/WEBEXT-2444

Rob Wu [:robwu]

Comment 5

•

6 months ago

Will, do you know how easy it is to get a recommended xpi file on stage? In this case, a line extension.

If difficult, then we can consider faking the recommendation in the test instead.

Flags: needinfo?(wdurand)

Comment hidden (Intermittent Failures Robot)

William Durand [:willdurand]

Assignee

Comment 7

•

6 months ago

Attached file Bug 1989205 - Update recommended.xpi file to fix a test (permafailure). r?robwu! — Details

The new XPI file has been generated from this PR 1, and signed by the
AMO -dev instance (which uses the nonprod cert). FTR since this add-on
needs a recommendation state, we must add the add-on to the "By Firefox"
promoted group via the AMO admin before getting the version signed.

Before:

$ xpidump old.xpi
MANIFEST:
  ID        : N/A
  Version   : 2.0
  Enterprise: No

RECOMMENDATION:
  [Line]

SIGNATURES:
  PKCS7:
   └── PRESENT / PRODUCTION / SHA-1 / REGULAR ADD-ON
   └── Certificates:
        └── Common Name         (CN): signingca1.addons.mozilla.org
            Organizational Unit (OU): Mozilla AMO Production Signing Service
            End Date                : 2025-04-04T00:00:00Z
        └── Common Name         (CN): {811d77f1-f306-4187-9251-b4ff99bad60b}
            Organizational Unit (OU): Production
            End Date                : 2030-09-15T16:50:45Z
  COSE:
   └── PRESENT / PRODUCTION / ES256 / REGULAR ADD-ON
   └── Certificates:
        └── Common Name         (CN): signingca1.addons.mozilla.org
            Organizational Unit (OU): Mozilla AMO Production Signing Service
            End Date                : 2025-04-04T00:00:00Z
        └── Common Name         (CN): {811d77f1-f306-4187-9251-b4ff99bad60b}
            Organizational Unit (OU): Production
            End Date                : 2030-09-15T16:50:45Z

After:

$ xpidump new.xpi
MANIFEST:
  ID        : recommended-line@test.mozilla.org
  Version   : 3.0
  Enterprise: No

RECOMMENDATION:
  [Line]

SIGNATURES:
  PKCS7:
   └── PRESENT / STAGING / SHA-256 / REGULAR ADD-ON
   └── Certificates:
        └── Common Name         (CN): cas-intermediate-amo-ca-staging
            Organizational Unit (OU): Mozilla Staging Signing Service
            End Date                : 2050-12-31T00:00:00Z
        └── Common Name         (CN): recommended-line@test.mozilla.org
            Organizational Unit (OU): Production
            End Date                : 2035-09-17T07:29:49Z
  COSE:
   └── PRESENT / STAGING / ES256 / REGULAR ADD-ON
   └── Certificates:
        └── Common Name         (CN): cas-intermediate-amo-ca-staging
            Organizational Unit (OU): Mozilla Staging Signing Service
            End Date                : 2050-12-31T00:00:00Z
        └── Common Name         (CN): recommended-line@test.mozilla.org
            Organizational Unit (OU): Production
            End Date                : 2035-09-17T07:29:49Z

Phabricator Automation

Updated

•

6 months ago

Assignee: nobody → wdurand

Status: NEW → ASSIGNED

William Durand [:willdurand]

Assignee

Updated

•

6 months ago

Flags: needinfo?(wdurand)

Rob Wu [:robwu]

Comment 8

•

6 months ago

I verified that this recommended.xpi file is the only xpi file with the mozilla-recommendation.json metadata file in the tree:

$ find . -name '*.xpi' | grep -v objdir- | while read line ; do [[ "$(zipinfo "$line" 2>/dev/null)" == *"mozilla-recommendation.json"* ]] && echo $line ; done
./toolkit/mozapps/extensions/test/xpinstall/recommended.xpi

William Durand [:willdurand]

Assignee

Comment 9

•

6 months ago

•

Edited

Comment on attachment 9514249 [details]
Bug 1989205 - Update recommended.xpi file to fix a test (permafailure). r?robwu!

Beta/Release Uplift Approval Request

User impact if declined/Reason for urgency: None (it's a test-only change).
Is this code covered by automated tests?: Yes
Has the fix been verified in Nightly?: No
Needs manual test from QE?: No
If yes, steps to reproduce:
List of other uplifts needed: None
Risk to taking this patch: Low
Why is the change risky/not risky? (and alternatives if risky): This patch replaces an XPI in tree to fix a permafailure. It can only get better...
String changes made/needed: N/A
Is Android affected?: Yes

Attachment #9514249 - Flags: approval-mozilla-release?

Attachment #9514249 - Flags: approval-mozilla-beta?

Pulsebot

Comment 10

•

6 months ago

Pushed by wdurand@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/3439e53eecf8 https://hg.mozilla.org/integration/autoland/rev/ba53bb716eda Update recommended.xpi file to fix a test (permafailure). r=robwu

Sandor Molnar[:smolnar]

Comment 11

•

6 months ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/ba53bb716eda

Status: ASSIGNED → RESOLVED

Closed: 6 months ago

status-firefox145: affected → fixed

Resolution: --- → FIXED

Target Milestone: --- → 145 Branch

Donal Meehan [:dmeehan]

Comment 12

•

6 months ago

Comment on attachment 9514249 [details]
Bug 1989205 - Update recommended.xpi file to fix a test (permafailure). r?robwu!

Approved for 144.0b4

Attachment #9514249 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Pulsebot

Comment 13

•

6 months ago

uplift

https://github.com/mozilla-firefox/firefox/commit/e30e88fc5c01 https://hg.mozilla.org/releases/mozilla-beta/rev/26f62b8e8519

Donal Meehan [:dmeehan]

Updated

•

6 months ago

status-firefox144: affected → fixed

Comment hidden (Intermittent Failures Robot)

William Durand [:willdurand]

Assignee

Comment 16

•

6 months ago

Attached file Bug 1989205 - Update recommended.xpi file to fix a test (permafailure) for ESR. r?robwu! — Details

The new XPI file has been generated from this PR 1, and signed by the
AMO -dev instance (which uses the nonprod cert). FTR since this add-on
needs a recommendation state, we must add the add-on to the "By Firefox"
promoted group via the AMO admin before getting the version signed.

Before:

$ xpidump old.xpi
MANIFEST:
  ID        : N/A
  Version   : 2.0
  Enterprise: No

RECOMMENDATION:
  [Line]

SIGNATURES:
  PKCS7:
    └── PRESENT / PRODUCTION / SHA-1 / REGULAR ADD-ON
    └── Certificates:
	└── Common Name         (CN): signingca1.addons.mozilla.org
	    Organizational Unit (OU): Mozilla AMO Production Signing Service
	    End Date                : 2025-04-04T00:00:00Z
	└── Common Name         (CN): {811d77f1-f306-4187-9251-b4ff99bad60b}
	    Organizational Unit (OU): Production
	    End Date                : 2030-09-15T16:50:45Z
  COSE:
    └── PRESENT / PRODUCTION / ES256 / REGULAR ADD-ON
    └── Certificates:
	└── Common Name         (CN): signingca1.addons.mozilla.org
	    Organizational Unit (OU): Mozilla AMO Production Signing Service
	    End Date                : 2025-04-04T00:00:00Z
	└── Common Name         (CN): {811d77f1-f306-4187-9251-b4ff99bad60b}
	    Organizational Unit (OU): Production
	    End Date                : 2030-09-15T16:50:45Z

After:

$ xpidump new.xpi
MANIFEST:
  ID        : recommended-line@test.mozilla.org
  Version   : 3.0
  Enterprise: No

RECOMMENDATION:
  [Line]

SIGNATURES:
  PKCS7:
    └── PRESENT / STAGING / SHA-256 / REGULAR ADD-ON
    └── Certificates:
	└── Common Name         (CN): cas-intermediate-amo-ca-staging
	    Organizational Unit (OU): Mozilla Staging Signing Service
	    End Date                : 2050-12-31T00:00:00Z
	└── Common Name         (CN): recommended-line@test.mozilla.org
	    Organizational Unit (OU): Production
	    End Date                : 2035-09-17T07:29:49Z
  COSE:
    └── PRESENT / STAGING / ES256 / REGULAR ADD-ON
    └── Certificates:
	└── Common Name         (CN): cas-intermediate-amo-ca-staging
	    Organizational Unit (OU): Mozilla Staging Signing Service
	    End Date                : 2050-12-31T00:00:00Z
	└── Common Name         (CN): recommended-line@test.mozilla.org
	    Organizational Unit (OU): Production
	    End Date                : 2035-09-17T07:29:49Z

Phabricator Automation

Updated

•

6 months ago

Attachment #9514638 - Flags: approval-mozilla-release?

Rob Wu [:robwu]

Comment 17

•

6 months ago

Comment on attachment 9514638 [details]
Bug 1989205 - Update recommended.xpi file to fix a test (permafailure) for ESR. r?robwu!

ESR Uplift Approval Request

If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fixes permafail
User impact if declined: None
Fix Landed on Version: 145, uplifted to beta (and pending uplift to release)
Risk to taking this patch: Low
Why is the change risky/not risky? (and alternatives if risky): Test-only fix of test that already permafails (timeout).

Attachment #9514638 - Flags: approval-mozilla-release?

Attachment #9514638 - Flags: approval-mozilla-esr140?

Attachment #9514638 - Flags: approval-mozilla-esr115?

Ryan VanderMeulen (PTO, back 6-April)

Comment 18

•

6 months ago

Comment on attachment 9514249 [details]
Bug 1989205 - Update recommended.xpi file to fix a test (permafailure). r?robwu!

test-only fix, approved for 143.0.3

Attachment #9514249 - Flags: approval-mozilla-release? → approval-mozilla-release+

Ryan VanderMeulen (PTO, back 6-April)

Updated

•

6 months ago

status-firefox143: affected → fixed

Pulsebot

Comment 19

•

6 months ago

uplift

https://github.com/mozilla-firefox/firefox/commit/a720515bb43d https://hg.mozilla.org/releases/mozilla-release/rev/1862d390f262

Phabricator Automation

Updated

•

6 months ago

Attachment #9514638 - Flags: approval-mozilla-esr140? → approval-mozilla-esr140+

Lando Automation

Updated

•

6 months ago

status-firefox-esr140: --- → fixed

Pulsebot

Comment 20

•

6 months ago

uplift

https://hg.mozilla.org/releases/mozilla-esr140/rev/38d1ea89c664

Ryan VanderMeulen (PTO, back 6-April)

Comment 21

•

6 months ago

Comment on attachment 9514638 [details]
Bug 1989205 - Update recommended.xpi file to fix a test (permafailure) for ESR. r?robwu!

Approved for 115.29esr.

Attachment #9514638 - Flags: approval-mozilla-esr115? → approval-mozilla-esr115+

Ryan VanderMeulen (PTO, back 6-April)

Updated

•

6 months ago

status-firefox-esr115: --- → fixed

Pulsebot

Comment 22

•

6 months ago

uplift

https://hg.mozilla.org/releases/mozilla-esr115/rev/f42a8130bb53

Mihai Boldan, Desktop QA [:mboldan]

Updated

•

5 months ago

QA Whiteboard: [qa-triage-done-c145/b144]

Serban Stanca [:SerbanS]

Updated

•

5 months ago

Bug 1989205 - Update recommended.xpi file to fix a test (permafailure). r?robwu! 6 months ago William Durand [:willdurand] 48 bytes, text/x-phabricator-request	dmeehan : approval-mozilla-beta+ ryanvm : approval-mozilla-release+	Details \| Review
Bug 1989205 - Update recommended.xpi file to fix a test (permafailure) for ESR. r?robwu! 6 months ago William Durand [:willdurand] 48 bytes, text/x-phabricator-request	ryanvm : approval-mozilla-esr115+ phab-bot : approval-mozilla-esr140+	Details \| Review

	msix	no_variant	spi-nw	swr
linux1804-x86_64/asan				2
linux1804-x86_64/debug				2
linux1804-x86_64/opt				2
linux1804-x86_64/tsan				4
linux2404-x86_64/asan				20
linux2404-x86_64/debug			8	22
linux2404-x86_64/tsan				12
macosx1015-x86_64/debug		20	8
macosx1015-x86_64/opt		22	14
macosx1500-aarch64/opt			2
windows11-x86/debug		12
windows11-x86/opt		14
windows11-x86_64/asan		16
windows11-x86_64/ccov		2
windows11-x86_64/debug	11	10	7
windows11-x86_64/opt	9	18	12

	msix	no_variant	spi-nw	swr
linux1804-x86_64/asan				2
linux1804-x86_64/debug				2
linux1804-x86_64/opt				2
linux1804-x86_64/tsan				2
linux2404-x86_64/asan				6
linux2404-x86_64/debug			4	6
linux2404-x86_64/tsan				5
macosx1015-x86_64/debug		7	3
macosx1015-x86_64/opt		11	4
macosx1470-x86_64/debug		1
macosx1500-aarch64/opt			1
windows11-x86/debug		9
windows11-x86/opt		7
windows11-x86_64/asan		6
windows11-x86_64/ccov		1
windows11-x86_64/debug	8	9	2
windows11-x86_64/opt	7	10	3

	msix	no_variant	spi-nw	swr
linux1804-x86_64/asan				4
linux1804-x86_64/debug				4
linux1804-x86_64/opt				4
linux1804-x86_64/tsan				6
linux2404-x86_64/asan				26
linux2404-x86_64/debug			12	28
linux2404-x86_64/tsan				17
macosx1015-x86_64/debug		27	11
macosx1015-x86_64/opt		33	18
macosx1470-x86_64/debug		1
macosx1500-aarch64/opt			3
windows11-x86/debug		21
windows11-x86/opt		21
windows11-x86_64/asan		23
windows11-x86_64/ccov		3
windows11-x86_64/debug	19	19	9
windows11-x86_64/opt	16	28	15

Permafail across trees TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js | Test timed out -

Repository breakdown:

Table

For more details, see:

Beta/Release Uplift Approval Request

Repository breakdown:

Table

For more details, see:

Repository breakdown:

Table

For more details, see:

ESR Uplift Approval Request