Closed Bug 1776048 Opened 3 years ago Closed 3 months ago

Frequent toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug

Categories

(Firefox :: Launcher Process, defect, P3)

Product:
Firefox
Component:
Launcher Process
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
135 Branch
Tracking Flags:
Tracking Status
firefox-esr128 --- unaffected
firefox132 --- unaffected
firefox133 --- wontfix
firefox134 --- wontfix
firefox135 --- fixed

People

(Reporter: jmaher, Assigned: rkraesig)

References

(Regression)

Details

(Keywords: intermittent-failure, intermittent-testcase, regression, Whiteboard: [stockwell disabled])

Attachments

(3 files, 1 obsolete file)

Bug 1776048 - disable browser_aboutthirdparty.js for frequent failures on windows. r=#intermittent-reviewers
48 bytes, text/x-phabricator-request
Details | Review
Bug 1776048 - disable aboutthirdparty test for debug builds to cut down on intermittency r=rkraesig
48 bytes, text/x-phabricator-request
Details | Review
Bug 1776048 - [1/1] Remove custom filepicker from test code r?gstoll!
48 bytes, text/x-phabricator-request
Details | Review
No description provided.

Additional information about this bug failures and frequency patterns can be found by running: ./mach test-info failure-report --bug 1776048

There have been 46 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-shippable-qr
  • windows10-64-2004-qr
  • windows7-32-qr
  • windows7-32-shippable-qr
[task 2022-11-10T22:49:13.245Z] 22:49:13     INFO - TEST-START | toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js
[task 2022-11-10T22:49:58.258Z] 22:49:58     INFO - TEST-INFO | started process screenshot
[task 2022-11-10T22:49:58.317Z] 22:49:58     INFO - TEST-INFO | screenshot: exit 0
[task 2022-11-10T22:49:58.317Z] 22:49:58     INFO - Buffered messages logged at 22:49:13
[task 2022-11-10T22:49:58.318Z] 22:49:58     INFO - Entering test bound 
[task 2022-11-10T22:49:58.318Z] 22:49:58     INFO - TEST-PASS | toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | The module file exists. - true == true - 
[task 2022-11-10T22:49:58.319Z] 22:49:58     INFO - registerObject() done - Z:\task_1668118444\build\tests\mochitest\browser\toolkit\components\aboutthirdparty\tests\browser\TestShellEx.dll
[task 2022-11-10T22:49:58.320Z] 22:49:58     INFO - Buffered messages logged at 22:49:15
[task 2022-11-10T22:49:58.320Z] 22:49:58     INFO - TEST-PASS | toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | lookupModuleType() returns a correct type after system info was collected. - 4 == 4 - 
[task 2022-11-10T22:49:58.321Z] 22:49:58     INFO - Buffered messages finished
[task 2022-11-10T22:49:58.322Z] 22:49:58     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | Test timed out - 
[task 2022-11-10T22:49:58.322Z] 22:49:58     INFO - GECKO(1352) | MEMORY STAT | vsize 996MB | vsizeMaxContiguous 512MB | residentFast 234MB | heapAllocated 199MB
[task 2022-11-10T22:49:58.323Z] 22:49:58     INFO - TEST-OK | toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | took 45030ms
[task 2022-11-10T22:49:58.323Z] 22:49:58     INFO - Not taking screenshot here: see the one that was previously logged
[task 2022-11-10T22:49:58.324Z] 22:49:58     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | Found a tab after previous test timed out: about:third-party - 
[task 2022-11-10T22:49:58.325Z] 22:49:58     INFO - checking window state
[task 2022-11-10T22:49:58.340Z] 22:49:58     INFO - GECKO(1352) | Completed ShutdownLeaks collections in process 5920
[task 2022-11-10T22:49:58.342Z] 22:49:58     INFO - TEST-START | Shutdown
[task 2022-11-10T22:49:58.343Z] 22:49:58     INFO - Browser Chrome Test Summary
[task 2022-11-10T22:49:58.343Z] 22:49:58     INFO - Passed:  2
[task 2022-11-10T22:49:58.343Z] 22:49:58     INFO - Failed:  2
[task 2022-11-10T22:49:58.344Z] 22:49:58     INFO - Todo:    0
[task 2022-11-10T22:49:58.344Z] 22:49:58     INFO - Mode:    e10s
[task 2022-11-10T22:49:58.345Z] 22:49:58     INFO - *** End BrowserChrome Test Results ***
[task 2022-11-10T22:49:58.402Z] 22:49:58     INFO - TEST-INFO | Main app process: exit 0
Flags: needinfo?(haftandilian)
Whiteboard: [stockwell needswork:owner]
Summary: Intermittent toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug → Permafailing toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug
Summary: Permafailing toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug → Almost permafailing toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug

The new permafail looks like a mozharness issue. Looking into the right product/component.

[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING - Unable to start resource monitor: Traceback (most recent call last):
[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING -   File "Z:\task_1668174215\mozharness\mozharness\base\python.py", line 822, in _start_resource_monitoring
[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING -     self._resource_monitor.start()
[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING -   File "Z:\task_1668174215\build\venv\Lib\site-packages\mozsystemmonitor\resourcemonitor.py", line 300, in start
[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING -     self._process.start()
[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING -   File "c:\mozilla-build\python3\lib\multiprocessing\process.py", line 112, in start
[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING -     self._popen = self._Popen(self)
[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING -   File "c:\mozilla-build\python3\lib\multiprocessing\context.py", line 223, in _Popen
[task 2022-11-11T14:08:27.520Z] 14:08:27  WARNING -     return _default_context.get_context().Process._Popen(process_obj)
[task 2022-11-11T14:08:27.521Z] 14:08:27  WARNING -   File "c:\mozilla-build\python3\lib\multiprocessing\context.py", line 322, in _Popen
[task 2022-11-11T14:08:27.521Z] 14:08:27  WARNING -     return Popen(process_obj)
[task 2022-11-11T14:08:27.521Z] 14:08:27  WARNING -   File "c:\mozilla-build\python3\lib\multiprocessing\popen_spawn_win32.py", line 72, in __init__
[task 2022-11-11T14:08:27.521Z] 14:08:27  WARNING -     env, None, False, 0, None, None, None)
[task 2022-11-11T14:08:27.521Z] 14:08:27  WARNING - FileNotFoundError: [WinError 2] The system cannot find the file specified

not sire if this would be fixed or changed by bug 1800055 (just landed on autoland today).

Do we know which file is not found?

Flags: needinfo?(haftandilian)
See Also: → 1800055

Note that the FileNotFoundError is actually just a warning: mozharness is failing to collect resource usage stats. That will be fixed by bug 1800055. But that does not turn the job orange in treeherder; the job is failing because of the TEST-UNEXPECTED-FAILs on browser_aboutthirdparty.js, and that's unrelated to the mozharness problem.

Assignee: nobody → csabou
Status: NEW → ASSIGNED
Pushed by csabou@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/8d8ef6442f44 disable browser_aboutthirdparty.js for frequent failures on windows. r=intermittent-reviewers,jmaher DONTBUILD
Assignee: csabou → nobody
Status: ASSIGNED → NEW
Summary: Almost permafailing toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug → Intermittent toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug
Whiteboard: [stockwell needswork:owner] → [stockwell disabled]
Keywords: leave-open
Attachment #9385034 - Attachment is obsolete: true

The original bug here was effectively fixed (by aging out all relevant Firefox versions), but not closed by the intermittent bug filer... and something completely different seems to have broken this test again circa 2024-10-02.

Most likely this was bug 1920545, which reenabled this test for most builds without apparent justification.... although really, that did no more than make the failure visible. Pinging :jmaher for context.

Digging into the failures themselves suggests that the test DLL just isn't showing up sometimes. I suspect this is due to a race condition in the terrifyingly hacky AboutThirdParty::OpenAndCloseFileDialogForTesting.

:gstoll, do you know of any reliable alternate ways to get a third-party DLL loaded into Firefox for testing (i.e., without involving opening a file picker)?

Flags: needinfo?(jmaher)
Flags: needinfo?(gstoll)
Keywords: regression
Regressed by: 1920545

Set release status flags based on info from the regressing bug 1920545

status-firefox132: --- → unaffected
status-firefox133: --- → affected
status-firefox134: --- → affected
status-firefox-esr128: --- → unaffected

thanks for flagging me on this. I had tested on try with 10 runs and never saw more than 2 failures so I assumed it was intermittent.

Looking at the 80 failures in the last 30 days 50+ are on ccov. debug seems to be about 1 failure/day and opt seems about 2 failures/day. I admit to not testing on ccov, so this was my oversight in that bug.

it looks like opt is also failing more frequently, we should consider investigating that or disabling there. At the very least debug seems to fail very infrequently.

Flags: needinfo?(jmaher)

I'm fine with this just running on debug builds, I'll make that change. Thanks!

Flags: needinfo?(gstoll)
Pushed by gstoll@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a318fa4b0e72 disable aboutthirdparty test for debug builds to cut down on intermittency r=rkraesig
Summary: Intermittent toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug → Frequent toolkit/components/aboutthirdparty/tests/browser/browser_aboutthirdparty.js | single tracking bug

Hi Ray, this started to also fail frequently on Windows 11 x64 22H2 WebRender Shippable , these fails started to spike on 3/12/2024, but the backfill range didn't point to an actual push that might have caused this, could you please assign it to someone?

Flags: needinfo?(rkraesig)

I'll see if I can't pick this up myself.

In the meantime, reiterating the question from comment 69 (which apparently got lost) — :gstoll, do you know of any reliable alternate ways to get a third-party DLL loaded into Firefox for testing (i.e., without involving a file picker)? In particular, would a straight ::LoadLibraryExW work?

Flags: needinfo?(gstoll)

Apologies, I looked at this and forgot to respond.

I think just calling ::LoadLibraryExW() would work? It certainly sounds more straightforward than opening a file picker...

Flags: needinfo?(gstoll)

The existing code for the browser_aboutthirdparty.js test attempts to
load a DLL indirectly by means of a Rube Goldberg process involving a
directly-instantiated file-picker. This appears to be the source of the
severe flakiness of this test.

This approach was reasonable when the file-picker was the primary source
of deleterious third-party DLLs; but now that we generally open file-
pickers out-of-process, there isn't as much of a reason to test that
particular injection mechanism. Let's just load the test DLL directly,
instead.

This patch has two caveats:

  • The test fails consistently, rather than flakily, on ASAN builds.
    The reason for that remains to be investigated, but probably belongs
    in a separate bug; for now, we simply leave it disabled.

  • The test expects the DLL to be a shell extension, and fails with
    complaints if it's not. There's no other reason to have any of the
    shell-extension-registration machinery in there, though, so it can
    (and probably should) be removed, and the test rewritten accordingly.
    In the interests of swiftly deflaking it, though, this is also being
    left for a separate bug.

Outside of test-only code, no functional changes.

Flags: needinfo?(rkraesig)
Pushed by rkraesig@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/dfa223ffb64b [1/1] Remove custom filepicker from test code r=gstoll

All listed failures on autoland or mozilla-central occurred on 2024-12-09, on builds which did not yet include D231462. (In fact, the last failures to occur on autoland and mozilla-central were for the same build — the one immediately before the fix.)

That's good enough for me. Closing as RESOLVED FIXED.

Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
status-firefox133: affected → wontfix
status-firefox134: affected → wontfix
status-firefox135: --- → fixed
Target Milestone: --- → 135 Branch
Assignee: nobody → rkraesig
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: