Closed Bug 1678102 Opened 4 years ago Closed 3 years ago

Intermittent browser/components/extensions/test/browser/browser_ext_menus.js | Test timed out | Extension left running at test shutdown -

Categories

(Core :: Layout, defect)

defect

Tracking

()

RESOLVED FIXED
91 Branch
Tracking Status
firefox-esr78 --- unaffected
firefox88 --- unaffected
firefox89 --- unaffected
firefox90 --- fixed
firefox91 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: mattwoodrow)

References

(Regression)

Details

(Keywords: intermittent-failure, regression, Whiteboard: [stockwell disable-recommended])

Attachments

(4 files, 1 obsolete file)

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=322209178&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fZ3ts51oR4uDpxbsRONqfA/runs/0/artifacts/public/logs/live_backing.log


[task 2020-11-18T18:33:21.465Z] 18:33:21     INFO - TEST-START | browser/components/extensions/test/browser/browser_ext_menus.js
[task 2020-11-18T18:34:06.485Z] 18:34:06     INFO - TEST-INFO | started process screencapture
[task 2020-11-18T18:34:06.592Z] 18:34:06     INFO - TEST-INFO | screencapture: exit 0
[task 2020-11-18T18:34:06.592Z] 18:34:06     INFO - Buffered messages logged at 18:33:21
[task 2020-11-18T18:34:06.593Z] 18:34:06     INFO - Entering test bound test_permissions
[task 2020-11-18T18:34:06.593Z] 18:34:06     INFO - Extension loaded
[task 2020-11-18T18:34:06.593Z] 18:34:06     INFO - Extension loaded
[task 2020-11-18T18:34:06.593Z] 18:34:06     INFO - Console message: Warning: attempting to write 12710 bytes to preference extensions.webextensions.uuids. This is bad for general performance and memory usage. Such an amount of data should rather be written to an external file. This preference will not be sent to any content processes.
[task 2020-11-18T18:34:06.593Z] 18:34:06     INFO - Console message: Warning: attempting to write 12790 bytes to preference extensions.webextensions.uuids. This is bad for general performance and memory usage. Such an amount of data should rather be written to an external file. This preference will not be sent to any content processes.
[task 2020-11-18T18:34:06.593Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | browser.menus available with 'menus' permission - 
[task 2020-11-18T18:34:06.593Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | browser.contextMenus unavailable with  'menus' permission - 
[task 2020-11-18T18:34:06.593Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | browser.menusInternal is never available - 
[task 2020-11-18T18:34:06.595Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | browser.menus unavailable with 'contextMenus' permission - 
[task 2020-11-18T18:34:06.595Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | browser.contextMenus unavailable with  'contextMenus' permission - 
[task 2020-11-18T18:34:06.595Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | browser.menusInternal is never available - 
[task 2020-11-18T18:34:06.595Z] 18:34:06     INFO - Leaving test bound test_permissions
[task 2020-11-18T18:34:06.599Z] 18:34:06     INFO - Entering test bound test_actionContextMenus
[task 2020-11-18T18:34:06.599Z] 18:34:06     INFO - Extension loaded
[task 2020-11-18T18:34:06.599Z] 18:34:06     INFO - Console message: Warning: attempting to write 12870 bytes to preference extensions.webextensions.uuids. This is bad for general performance and memory usage. Such an amount of data should rather be written to an external file. This preference will not be sent to any content processes.
[task 2020-11-18T18:34:06.605Z] 18:34:06     INFO - Console message: Warning: attempting to write 6646 bytes to preference browser.uiCustomization.state. This is bad for general performance and memory usage. Such an amount of data should rather be written to an external file. This preference will not be sent to any content processes.
[task 2020-11-18T18:34:06.606Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct submenu type - 
[task 2020-11-18T18:34:06.606Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct submenu title - 
[task 2020-11-18T18:34:06.606Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct submenu opened - 
[task 2020-11-18T18:34:06.606Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct number of submenu items - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Second menu item type is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Second menu item title is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Second menu item id is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Last menu item type is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Last menu item title is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Last menu ext-type is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Separator after last menu item - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - Buffered messages logged at 18:33:22
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Excess items should be moved into a submenu - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | First submenu item ID is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Last submenu item ID is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Click info pageUrl is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Click event tab ID is correct - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct submenu type - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct submenu title - 
[task 2020-11-18T18:34:06.608Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct submenu opened - 
[task 2020-11-18T18:34:06.609Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct number of submenu items - 
[task 2020-11-18T18:34:06.609Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Second menu item type is correct - 
[task 2020-11-18T18:34:06.609Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Second menu item title is correct - 
[task 2020-11-18T18:34:06.609Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Second menu item id is correct - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Last menu item type is correct - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Last menu item title is correct - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Last menu ext-type is correct - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Separator after last menu item - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Excess items should be moved into a submenu - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | First submenu item ID is correct - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Last submenu item ID is correct - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - Buffered messages logged at 18:33:23
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Click info pageUrl is correct - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Click event tab ID is correct - 
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - Console message: [JavaScript Error: "sendRemoveListener on closed conduit {a79fd3cc-f150-d644-9755-a786fa68cdf7}.137438955603" {file: "resource://gre/modules/ConduitsChild.jsm" line: 108}]
[task 2020-11-18T18:34:06.615Z] 18:34:06     INFO - _send@resource://gre/modules/ConduitsChild.jsm:108:13
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - removeListener@resource://gre/modules/ExtensionChild.jsm:663:34
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - removeListener@resource://gre/modules/ExtensionChild.jsm:886:40
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - getAPI/register/api.menus.onClicked@chrome://browser/content/child/ext-menus.js:282:21
task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - removeListener@resource://gre/modules/ExtensionCommon.jsm:2520:7
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - revoke@resource://gre/modules/ExtensionCommon.jsm:2542:12
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - close@resource://gre/modules/ExtensionCommon.jsm:2547:10
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - unload@resource://gre/modules/ExtensionCommon.jsm:912:11
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - unload@resource://gre/modules/ExtensionPageChild.jsm:261:11
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - unload@resource://gre/modules/ExtensionPageChild.jsm:298:11
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - destroyExtensionContext@resource://gre/modules/ExtensionPageChild.jsm:482:15
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - observe@resource://gre/modules/ExtensionPageChild.jsm:395:12
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - 
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - Leaving test bound test_actionContextMenus
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - Entering test bound test_hiddenPageActionContextMenu
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - Extension loaded
[task 2020-11-18T18:34:06.616Z] 18:34:06     INFO - Console message: Warning: attempting to write 12950 bytes to preference extensions.webextensions.uuids. This is bad for general performance and memory usage. Such an amount of data should rather be written to an external file. This preference will not be sent to any content processes.
[task 2020-11-18T18:34:06.617Z] 18:34:06     INFO - Buffered messages logged at 18:33:24
[task 2020-11-18T18:34:06.617Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct number of children - 
[task 2020-11-18T18:34:06.617Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct first child - 
[task 2020-11-18T18:34:06.617Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct second child - 
[task 2020-11-18T18:34:06.619Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct third child - 
[task 2020-11-18T18:34:06.619Z] 18:34:06     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Correct fourth child - 
[task 2020-11-18T18:34:06.620Z] 18:34:06     INFO - Leaving test bound test_hiddenPageActionContextMenu
[task 2020-11-18T18:34:06.620Z] 18:34:06     INFO - Entering test bound test_bookmarkContextMenu
[task 2020-11-18T18:34:06.620Z] 18:34:06     INFO - Extension loaded
[task 2020-11-18T18:34:06.620Z] 18:34:06     INFO - Console message: Warning: attempting to write 13030 bytes to preference extensions.webextensions.uuids. This is bad for general performance and memory usage. Such an amount of data should rather be written to an external file. This preference will not be sent to any content processes.
[task 2020-11-18T18:34:06.620Z] 18:34:06     INFO - Buffered messages finished
[task 2020-11-18T18:34:06.621Z] 18:34:06     INFO - TEST-UNEXPECTED-FAIL | browser/components/extensions/test/browser/browser_ext_menus.js | Test timed out - 
[task 2020-11-18T18:34:06.621Z] 18:34:06     INFO - Not taking screenshot here: see the one that was previously logged
[task 2020-11-18T18:34:06.621Z] 18:34:06     INFO - TEST-UNEXPECTED-FAIL | browser/components/extensions/test/browser/browser_ext_menus.js | Extension left running at test shutdown - 
[task 2020-11-18T18:34:06.621Z] 18:34:06     INFO - Stack trace:
[task 2020-11-18T18:34:06.622Z] 18:34:06     INFO - chrome://mochikit/content/browser-test.js:test_ok:1304
[task 2020-11-18T18:34:06.622Z] 18:34:06     INFO - chrome://mochikit/content/tests/SimpleTest/ExtensionTestUtils.js:ExtensionTestUtils.loadExtension/<:117
[task 2020-11-18T18:34:06.622Z] 18:34:06     INFO - chrome://mochikit/content/browser-test.js:nextTest:555
[task 2020-11-18T18:34:06.623Z] 18:34:06     INFO - GECKO(1533) | MEMORY STAT | vsize 8607MB | residentFast 749MB | heapAllocated 184MB
[task 2020-11-18T18:34:06.625Z] 18:34:06     INFO - TEST-OK | browser/components/extensions/test/browser/browser_ext_menus.js | took 45087ms
[task 2020-11-18T18:34:06.625Z] 18:34:06     INFO - checking window state```
Summary: Intermittent browser/components/extensions/test/browser/browser_ext_menus.js | Extension left running at test shutdown - → Intermittent browser/components/extensions/test/browser/browser_ext_menus.js | Test timed out | Extension left running at test shutdown -

Update:

There has been a total of 57 failures within the last 7 days:

  • 23 failures on Linux 18.04 x64 shippable opt
  • 4 failures on Linux 18.04 x64 CCov opt
  • 1 failure on Linux 18.04 x64 asan opt
  • 29 failures on Linux 18.04 x64 opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=339594498&repo=mozilla-central&lineNumber=6027

[task 2021-05-13T10:38:51.824Z] 10:38:51     INFO - TEST-PASS | browser/components/extensions/test/browser/browser_ext_menus.js | Click event tab ID is correct - 
[task 2021-05-13T10:38:51.824Z] 10:38:51     INFO - Buffered messages finished
[task 2021-05-13T10:38:51.826Z] 10:38:51     INFO - TEST-UNEXPECTED-FAIL | browser/components/extensions/test/browser/browser_ext_menus.js | Test timed out - 
[task 2021-05-13T10:38:51.827Z] 10:38:51     INFO - Not taking screenshot here: see the one that was previously logged
[task 2021-05-13T10:38:51.828Z] 10:38:51     INFO - TEST-UNEXPECTED-FAIL | browser/components/extensions/test/browser/browser_ext_menus.js | Extension left running at test shutdown - 
[task 2021-05-13T10:38:51.828Z] 10:38:51     INFO - Stack trace:
[task 2021-05-13T10:38:51.828Z] 10:38:51     INFO - chrome://mochikit/content/browser-test.js:test_ok:1334
[task 2021-05-13T10:38:51.828Z] 10:38:51     INFO - chrome://mochikit/content/tests/SimpleTest/ExtensionTestUtils.js:ExtensionTestUtils.loadExtension/<:117
[task 2021-05-13T10:38:51.828Z] 10:38:51     INFO - chrome://mochikit/content/browser-test.js:nextTest:564
[task 2021-05-13T10:38:51.830Z] 10:38:51     INFO - Console message: [JavaScript Error: "sendRemoveListener on closed conduit {4782ee86-a473-4842-912f-331fb0f3bc38}.274877909007" {file: "resource://gre/modules/ConduitsChild.jsm" line: 108}]
[task 2021-05-13T10:38:51.830Z] 10:38:51     INFO - _send@resource://gre/modules/ConduitsChild.jsm:108:13
[task 2021-05-13T10:38:51.830Z] 10:38:51     INFO - removeListener@resource://gre/modules/ExtensionChild.jsm:667:34
[task 2021-05-13T10:38:51.830Z] 10:38:51     INFO - removeListener@resource://gre/modules/ExtensionChild.jsm:890:40
[task 2021-05-13T10:38:51.830Z] 10:38:51     INFO - getAPI/register/api.menus.onClicked@chrome://browser/content/child/ext-menus.js:284:21
[task 2021-05-13T10:38:51.830Z] 10:38:51     INFO - removeListener@resource://gre/modules/ExtensionCommon.jsm:2533:7
[task 2021-05-13T10:38:51.830Z] 10:38:51     INFO - revoke@resource://gre/modules/ExtensionCommon.jsm:2555:12
[task 2021-05-13T10:38:51.830Z] 10:38:51     INFO - close@resource://gre/modules/ExtensionCommon.jsm:2560:10
[task 2021-05-13T10:38:51.831Z] 10:38:51     INFO - unload@resource://gre/modules/ExtensionCommon.jsm:922:11
[task 2021-05-13T10:38:51.831Z] 10:38:51     INFO - unload@resource://gre/modules/ExtensionPageChild.jsm:261:11
[task 2021-05-13T10:38:51.831Z] 10:38:51     INFO - unload@resource://gre/modules/ExtensionPageChild.jsm:298:11
[task 2021-05-13T10:38:51.831Z] 10:38:51     INFO - destroyExtensionContext@resource://gre/modules/ExtensionPageChild.jsm:482:15
[task 2021-05-13T10:38:51.831Z] 10:38:51     INFO - observe@resource://gre/modules/ExtensionPageChild.jsm:395:12
[task 2021-05-13T10:38:51.831Z] 10:38:51     INFO - 
[task 2021-05-13T10:38:51.831Z] 10:38:51     INFO - GECKO(6282) | MEMORY STAT | vsize 4050MB | residentFast 720MB | heapAllocated 194MB
[task 2021-05-13T10:38:51.832Z] 10:38:51     INFO - TEST-OK | browser/components/extensions/test/browser/browser_ext_menus.js | took 45029ms
Flags: needinfo?(mixedpuppy)
Whiteboard: [stockwell needswork:owner]
Assignee: nobody → apavel
Status: NEW → ASSIGNED
Assignee: apavel → nobody
Status: ASSIGNED → NEW
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Assignee: nobody → apavel
Status: NEW → ASSIGNED

In my recent try pushes of unrelated patches, I frequently noticed failures in the browser_ext_menus_*.js tests (these tests were usually different). Disabling this one test doesn't fix the failures of the other tests, so I suggest to not land the patch to disable a test.

Most failures are just a "Test timed out" in the test_actionContextMenus test. I don't see errors, but the screenshot shows that the menu is stuck. I can reproduce the failure quite frequently locally, by isolating this test task and running it a few times (or running the test task a few times in the same test file).
When the test is stuck, the browser looks like this: https://firefoxci.taskcluster-artifacts.net/FoAFuqfkTG-QDPZpga88NQ/0/public/test_info/mozilla-test-fail-screenshot_wvjhzl82.png

I cannot interact with the context menu (at most the menu disappears on click, even though I expected a submenu to open). Arrow keys cannot be used. My guess is that opening the menu quickly after closing it previously is triggering this bug. This may be related to the recent refactor of the menu implementation for MR1. I'll take a look, because these failures are too frequent.
For now, I mainly want to know whether the test failures are an indication of a real bug or a test-only issue.

Assignee: apavel → rob
Flags: needinfo?(mixedpuppy)
Priority: P5 → P2

Screenshot of menu being stuck, for posterity.

I've observed two issues that contribute to this test timeout. It's possible that there is a different root cause, but this is what I see:

  1. The context menu is not closed despite the call to closeActionContextMenu. That function calls .activateItem, which was supposed to close the main menu when a submenu item was clicked. Replacing return hidden; with await hidden; menu.hidePopup(); would fix this, a work-around similar to bug 1351638
  2. Regardless of the fix for 1, the test is stuck at a requestAnimationFrame call, via a call to openActionContextMenu. I can open the console with Ctrl-Shift-J and confirm that calling requestAnimationFrame doesn't result in invocation of the callback.
  • Context menus, app menus are partially rendered. Hovering over menu items does not result in a visual hover effect (i.e. an updated background color).
  • When I open a new browser window (e.g. Ctrl-Shift-P), the UI works as expected. Opening the browser console for this window and calling requestAnimationFrame(console.log) results in the expected invocation of the callback.

Given the relevance of requestAnimationFrame, I looked up where the callback was supposed to be invoked, and found nsRefreshDriver.

MOZ_LOG=nsRefreshDriver:5 MOZCONFIG=.mozconfig-opt ./mach test browser/components/extensions/test/browser/browser_ext_menus.js

(for ease of reproduction, I removed everything from the test except for test_actionContextMenus at https://searchfox.org/mozilla-central/rev/2b372b94ce057097a6ef8eb725f209faa9d1dc4d/browser/components/extensions/test/browser/browser_ext_menus.js#65-153 , plus called that test task multiple times; --verify can also be used instead of duplicating the test task.)

When the submenu is shown, the following is logged:

[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5782984800] Completed transaction id 116
[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5782984800] Completed transaction id 115
[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5782984800] No longer over pending transaction limit, leaving wait state
[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5784a22ca0] ticking drivers...
[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5782984800] Allocating transaction id 120
[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5782984800] Hit max pending transaction limit, entering wait mode

In the non-broken case, I eventually see No longer over pending transaction limit, leaving wait state
But in the test stuck case (where requestAnimationFrame isn't processed any more), I see the logspam that repeats the following:

[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5782984800] ticking drivers...
[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5782984800] Over max pending transaction limit when trying to paint, skipping
[Parent 3682245: Main Thread]: D/nsRefreshDriver [7f5782984800] done.

This logging was introduced in bug 1708325.

Matt, does the log in comment 34 indicate a problem with nsRefreshDriver?
When this bug happens I observe several layout issues, and requestAnimationFrame's callbacks aren't called. Since the changes (and frequency of test failures) are quite recent, I'm wondering if this is a bug in nsRefreshDriver (or maybe an issue with refactored code to support native menus).

Flags: needinfo?(matt.woodrow)

FWIW, I'm fixing a different cause of failures in this test in https://phabricator.services.mozilla.com/D116118 .

Locally and in some logs on current central, I do see a number of exceptions in extension code in the console, along the lines of:

[task 2021-05-27T14:05:43.818Z] 14:05:43 INFO - GECKO(1543) | JavaScript error: chrome://browser/content/parent/ext-pageAction.js, line 268: TypeError: n.getAttribute is not a function

Full stack appears to be along the lines of:

 6:13.66 INFO Console message: [JavaScript Error: "TypeError: n.getAttribute is not a function" {file: "chrome://browser/content/parent/ext-pageAction.js" line: 268}]
handleEvent@chrome://browser/content/parent/ext-pageAction.js:268:26
synthesizeMouseAtPoint@chrome://mochikit/content/tests/SimpleTest/EventUtils.js:574:32
synthesizeMouse@chrome://mochikit/content/tests/SimpleTest/EventUtils.js:508:10
synthesizeMouseAtCenter@chrome://mochikit/content/tests/SimpleTest/EventUtils.js:681:10
openChromeContextMenu@chrome://mochitests/content/browser/browser/components/extensions/test/browser/head.js:567:14
openActionContextMenu@chrome://mochitests/content/browser/browser/components/extensions/test/browser/head.js:608:10

(the head.js line numbers may be slightly off due to the above-mentioned patch)

I don't know if this is related or a red herring.

(In reply to :Gijs (he/him) from comment #36)

FWIW, I'm fixing a different cause of failures in this test in https://phabricator.services.mozilla.com/D116118 .

Locally and in some logs on current central, I do see a number of exceptions in extension code in the console, along the lines of:

[task 2021-05-27T14:05:43.818Z] 14:05:43 INFO - GECKO(1543) | JavaScript error: chrome://browser/content/parent/ext-pageAction.js, line 268: TypeError: n.getAttribute is not a function

I noticed it before, I have a patch already and will upload it after my meetings. bug 1713120

Attachment #9222754 - Attachment is obsolete: true
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

I locally reverted the two patches from bug 1708325.

Hence I'm marking this as a regression of bug 1708325. Comment 34 has more logging / info that may be relevant for the investigation of this regression. I'll attach a reduced test case for ease of reproduction.

Keywords: regression
Regressed by: 1708325
Has Regression Range: --- → yes

STR:

  1. Add [browser_ext_menus_reduced_str.js] to browser/components/extensions/test/browser/browser.ini
  2. Run ./mach test browser/components/extensions/test/browser/browser_ext_menus_reduced_str.js . You may need to run this a couple of times. Alternatively, run in test verification mode (i.e. automatically run the test repeatedly) with:
    ./mach test browser/components/extensions/test/browser/browser_ext_menus_reduced_str.js --verify

For more logs (as seen in comment 34), prepend: MOZ_LOG=nsRefreshDriver:5

Expected:

  • Test runs to completion.

Actual:

  • Test gets stuck, context menu stays open, e.g. as seen at comment 33.

Reproduced on ArchLinux + KDE, locally built (artifact and non-artifact build) with checkout at https://hg.mozilla.org/mozilla-central/rev/4db2640cb0cd60103ffa75fa76b11fa66257774f.

EDIT: still happens with the latest tip of the tree at https://hg.mozilla.org/mozilla-central/rev/007ed77b9cf0e8b66de757b9f503cb2392f76872

Attached file log-full.log

Log output of a failing run, using STR from comment 41.
I created the following tool to analyze the log output, which looks up the matching "Revoked/Completed transaction id XXX" for an observed "Allocating transaction id XXX": https://jsfiddle.net/k3524onq/2/

In every failing test run, I find that there are entries where there is an allocation without "Revoked/Completed" message, followed by a few duplicate "Revoked"/"Completed" lines for the transactions that were created after, for example:

115	C
116	
117	C
118	CCC
119	RCC

116 should have had a C (or at least a R), 118 and 119 have too many C's. After additional debugging, I found that the nsRefreshDriver::NotifyTransactionCompleted call comes from one of these two places:

Actual log that was summarized above (even fuller log in attachment):

 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Allocating transaction id 115
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Hit max pending transaction limit, entering wait mode
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Allocating transaction id 116
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Allocating transaction id 117
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Allocating transaction id 118
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160d3e7f40] done.
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Allocating transaction id 119
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Revoking transaction id 119
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Allocating transaction id 120
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Revoking transaction id 120
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Allocating transaction id 121
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Revoking transaction id 121
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Allocating transaction id 122
 0:05.95 GECKO(765321) [Parent 765321: Main Thread]: D/nsRefreshDriver [7f160b38f000] Revoking transaction id 122

Kicking this bug to Core::Layout since this is not an issue with extension tests, but our tests exposed an issue with the changes to nsRefreshDriver from bug 1708325.

Core::Graphics::Layers could also be a good place for this bug since the trigger for the deadlock could be in the way that ClientLayerManager keeps track of mLatestTransactionId.

Assignee: rob → nobody
Severity: S4 → --
Status: ASSIGNED → NEW
Component: Untriaged → Layout
Priority: P2 → --
Product: WebExtensions → Core
Blocks: 1710981

Hi Rob, this has reached the disable-recommended queue. Can it be disabled until there is a fix or should we wait some more?

Flags: needinfo?(rob)

My recommendation is to revert the patches from bug 1708325 since that patch is provably the immediate cause of this regression (see comment 40). Ideally, before doing so I'd like to hear back from the patch author (or a reviewer, or anyone involved in that component), but haven't heard back from the needinfo here yet (commennt 35).

I'm not really familiar with what bug 1708325 hopes to achieve, but I can observe that it is causing this bug.
I'm afraid that the test failures are just a symptom of a larger issue.

Flags: needinfo?(rob)
Flags: needinfo?(mstange.moz)

@Markus can you please provide your input on the above?

Thank you @Rob!

I see, so we're not painting because we're waiting for a composite to finish. Context menus are painted from the main window refresh driver. But context menus are not composited by the compositor at the moment, I think, so we won't get a DidComposite notification for them. So I'm not sure how that case is handled. Matt, do you know?

I agree that somebody who knows the refresh driver needs to look into this. Given the steps to reproduce in comment 41 it hopefully shouldn't take too long to figure out what's going on.

Flags: needinfo?(mstange.moz)
See Also: → 1682681
See Also: → 1490052
Flags: needinfo?(matt.woodrow)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Flags: needinfo?(matt.woodrow)
Assignee: nobody → matt.woodrow
Flags: needinfo?(matt.woodrow)

I haven't been able to reproduce this on Ubuntu or Windows10.

Rob, does this patch fix the issue for you? https://treeherder.mozilla.org/jobs?repo=try&revision=853c34b39f8de0ac9331a7b91db56a51e9baaa44

Thanks for investigating this!

Flags: needinfo?(rob)

I'll have access to my Linux laptop in a few days, I'll give it a try as soon as I can.

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

(In reply to Matt Woodrow (:mattwoodrow) from comment #56)

I haven't been able to reproduce this on Ubuntu or Windows10.

Rob, does this patch fix the issue for you? https://treeherder.mozilla.org/jobs?repo=try&revision=853c34b39f8de0ac9331a7b91db56a51e9baaa44

Yep, it does! I checked out the tip of the tree and ran the STR from comment 41. It failed at the first run.
Then I applied your patch, rebuild Firefox and ran the test again. The test passed. Then I repeated with --verify and the test still passes.

Please submit the patch and request an uplift to 90.

Flags: needinfo?(rob)
Pushed by mwoodrow@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cef163a931a7
Store the set of pending transactions in ClientLayerManager, to ensure we clear all of them during Destroy. r=mstange
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 91 Branch

Comment on attachment 9226663 [details]
Bug 1678102 - Store the set of pending transactions in ClientLayerManager, to ensure we clear all of them during Destroy. r?mstange

Beta/Release Uplift Approval Request

  • User impact if declined: Browser UI may stop painting (and requestAnimationFrame callbacks will never be run) when a context menu is opened. This was a regression in Firefox 90, currently on beta.
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Straightforward fix. Manually verified in comment 59. Intermittent failures viewer has shown a significant drop (from many to zero) test failures in related tests.
  • String changes made/needed:
Attachment #9226663 - Flags: approval-mozilla-beta?

Comment on attachment 9226663 [details]
Bug 1678102 - Store the set of pending transactions in ClientLayerManager, to ensure we clear all of them during Destroy. r?mstange

approved for 90.0b9

Attachment #9226663 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: