1645563 - Intermittent /test/mochitest/browser_dbg-scopes-duplicated.js | A promise chain failed to handle a rejection: Can not send request 'getNodeActorFromObjectActor' because front 'domwalker' is already destroyed. - stack: generateRequestMethods

Status: RESOLVED FIXED

(DevTools :: Debugger, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=306190937&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SfwYWhMRSoeGomxHioZnrg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-06-13T02:40:00.798Z] 02:40:00 INFO - TEST-PASS | devtools/client/debugger/test/mochitest/browser_dbg-scopes-duplicated.js | The main process DevToolsServer has no pending connection when the test ends -
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - Console message: [JavaScript Error: "Error: Can not send request 'getNodeActorFromObjectActor' because front 'domwalker' is already destroyed." {file: "resource://devtools/shared/protocol/Front/FrontClassWithSpec.js" line: 29}]
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - Buffered messages finished
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - TEST-UNEXPECTED-FAIL | devtools/client/debugger/test/mochitest/browser_dbg-scopes-duplicated.js | A promise chain failed to handle a rejection: Can not send request 'getNodeActorFromObjectActor' because front 'domwalker' is already destroyed. - stack: generateRequestMethods/</frontProto[name]@resource://devtools/shared/protocol/Front/FrontClassWithSpec.js:29:15
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - gripToNodeFront@resource://devtools/client/fronts/walker.js:156:33
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - getNodeFrontFromNodeGrip@resource://devtools/client/fronts/inspector.js:163:26
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - getHighlighter/highlight/pendingHighlight<@resource://devtools/client/framework/toolbox.js:3529:46
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - async*highlight@resource://devtools/client/framework/toolbox.js:3539:11
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - highlightDomElement@resource://devtools/client/debugger/panel.js:157:17
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - highlightDomElement/<@resource://devtools/client/debugger/src/actions/toolbox.js:57:18
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - thunk/</</<@resource://devtools/client/debugger/src/actions/utils/middleware/thunk.js:29:45
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - bindActionCreator/<@resource://devtools/client/shared/vendor/redux.js:520:12
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - onDOMNodeMouseOver@resource://devtools/client/debugger/src/components/SecondaryPanes/Scopes.js:188:56
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - onMouseOver@resource://devtools/client/shared/components/reps/reps.js:7081:45
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - invokeGuardedCallbackImpl@resource://devtools/client/shared/vendor/react-dom.js:74:10
[task 2020-06-13T02:40:00.799Z] 02:40:00 INFO - invokeGuardedCallback@resource://devtools/client/shared/vendor/react-dom.js:111:29
[task 2020-06-13T02:40:00.800Z] 02:40:00 INFO - invokeGuardedCallbackAndCatchFirstError@resource://devtools/client/shared/vendor/react-dom.js:125:25
[task 2020-06-13T02:40:00.800Z] 02:40:00 INFO - executeDispatch@resource://devtools/client/shared/vendor/react-dom.js:346:42
[task 2020-06-13T02:40:00.800Z] 02:40:00 INFO - executeDispatchesInOrder@resource://devtools/client/shared/vendor/react-dom.js:365:20
[task 2020-06-13T02:40:00.800Z] 02:40:00 INFO - executeDispatchesAndRelease@resource://devtools/client/shared/vendor/react-dom.js:462:29
[task 2020-06-13T02:40:00.800Z] 02:40:00 INFO - executeDispatchesAndReleaseTopLevel@resource://devtools/client/shared/vendor/react-dom.js:470:10
[task 2020-06-13T02:40:00.800Z] 02:40:00 INFO - forEachAccumulated@resource://devtools/client/shared/vendor/react-dom.js:444:8
[task 2020-06-13T02:40:00.800Z] 02:40:00 INFO - runEventsInBatch@resource://devtools/client/shared/vendor/react-dom.js:598:21
[task 2020-06-13T02:40:00.800Z] 02:40:00 INFO - runExtractedEventsInBatch@resource://devtools/client/shared/vendor/react-dom.js:606:19

Comment 1

[Intermittent Failures Robot]

17 failures in 186 pushes (0.091 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 5
• try: 1

Platform breakdown:

• windows7-32: 5
• windows10-64: 10
• windows10-64-ccov: 1
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-13&endday=2020-06-13&tree=all

Comment 2

[Razvan Maries]

Based on https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=Windows%2C7%2Cdebug%2CMochitests%2Ctest-windows7-32%2Fdebug-mochitest-devtools-chrome-e10s-4%2CM%28dt4%29&fromchange=99ea9e6229cbb44a344afc83b572a0450777aac9&tochange=308399471b6e70a047951712e4cb8e91bfc795a8&selectedTaskRun=Gb96CyX_Qb2fg1dKNXjulw.0 I think bug 1644669 is the culprit.

Marco, could you take a look?

Comment 3

[Marco Zehe (:MarcoZ)]

My push from bug 1644669 didn't touch any of that code. So I don't see how this intermittent could be related. Someone more familiar with the Debugger tests should look at this.

Comment 4

[Razvan Maries]

Continued retriggers: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cdebug%2Cmochitests%2Ctest-windows7-32%2Fdebug-mochitest-devtools-chrome-e10s-4%2Cm%28dt4%29&fromchange=0b4d785bd7a16c1ccbac06957ee784d3f5c49750&tochange=308399471b6e70a047951712e4cb8e91bfc795a8

Comment 5

[Natalia Csoregi [:nataliaCs]]

Continued retriggers and backfills: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cdebug%2Cmochitests%2Ctest-windows7-32%2Fdebug-mochitest-devtools-chrome-e10s-4%2Cm%28dt4%29&fromchange=ca053f5829f0856d66291432ebfe55722bc7c066&tochange=308399471b6e70a047951712e4cb8e91bfc795a8&group_state=expanded&selectedTaskRun=Ts8WJexRSympnyP8RrR7Sg.0

Moving to another chunk found so far: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=Windows%2C10%2Cx64%2Casan%2CMochitests%2Ctest-windows10-64-asan%2Fopt-mochitest-devtools-chrome-e10s-8%2CM%28dt8%29&fromchange=ca053f5829f0856d66291432ebfe55722bc7c066&tochange=308399471b6e70a047951712e4cb8e91bfc795a8&selectedTaskRun=TAAa568KRjq4Al_OGZJolQ.0

Could it be Bug 1594467?

Comment 6

[Intermittent Failures Robot]

24 failures in 5211 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 17
• mozilla-central: 5
• try: 2

Platform breakdown:

• windows10-64: 17
• windows7-32: 5
• windows10-64-ccov: 1
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-08&endday=2020-06-14&tree=all

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Can you check what started this frequent failure, please?

Comment 8

[Julian Descottes [:jdescottes]]

aryx: Did you mean to close this one as duplicate? The two bugs don't seem related.

Comment 9

[Intermittent Failures Robot]

23 failures in 874 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18
• try: 3
• mozilla-central: 2

Platform breakdown:

• windows7-32: 5
• windows10-64: 17
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-16&endday=2020-06-16&tree=all

Comment 10

[Intermittent Failures Robot]

19 failures in 942 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 3
• try: 1

Platform breakdown:

• windows10-64: 14
• windows7-32-shippable: 2
• windows7-32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-17&endday=2020-06-17&tree=all

Comment 11

[Intermittent Failures Robot]

76 failures in 4035 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #15 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 58
• try: 7
• mozilla-central: 11

Platform breakdown:

• windows10-64: 54
• windows7-32: 15
• windows7-32-shippable: 5
• windows10-64-ccov: 1
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-15&endday=2020-06-21&tree=all

Comment 12

[Intermittent Failures Robot]

18 failures in 1020 pushes (0.018 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• mozilla-central: 4

Platform breakdown:

• windows10-64: 14
• windows7-32: 2
• windows10-64-shippable: 1
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-23&endday=2020-06-23&tree=all

Comment 13

[Intermittent Failures Robot]

21 failures in 1046 pushes (0.02 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 18
• mozilla-central: 3

Platform breakdown:

• windows10-64: 17
• windows10-64-shippable: 1
• windows7-32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-24&endday=2020-06-24&tree=all

Comment 14

[Intermittent Failures Robot]

23 failures in 1151 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• try: 1
• mozilla-central: 1

Platform breakdown:

• windows7-32: 8
• windows10-64: 14
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-25&endday=2020-06-25&tree=all

Comment 15

[Razvan Maries]

Julian, this bug is on our disable-recommended filter. Are there any updates?

Comment 16

[Razvan Maries]

Attachment: Bug 1645563 - Disabled browser_dbg-scopes-duplicated.js on Windows. r?#intermittent-reviewers

Comment 17

[Intermittent Failures Robot]

20 failures in 806 pushes (0.025 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 3
• try: 1

Platform breakdown:

• windows7-32: 3
• windows10-64: 14
• windows7-32-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-26&endday=2020-06-26&tree=all

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ca79c56a0a1b
Disabled browser_dbg-scopes-duplicated.js on Windows. r=jmaher

Comment 19

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/ca79c56a0a1b

Comment 20

[Intermittent Failures Robot]

106 failures in 5252 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 88
• mozilla-central: 14
• try: 4

Platform breakdown:

• windows10-64: 75
• windows7-32: 20
• windows10-64-ccov: 1
• windows7-32-shippable: 7
• windows10-64-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1645563&startday=2020-06-22&endday=2020-06-28&tree=all

Comment 21

[Julian Descottes [:jdescottes]]

Here's what I could gather. We can focus on this part of the stack:

gripToNodeFront@resource://devtools/client/fronts/walker.js:156:33
getNodeFrontFromNodeGrip@resource://devtools/client/fronts/inspector.js:163:26
getHighlighter/highlight/pendingHighlight<@resource://devtools/client/framework/toolbox.js:3529:46
async*highlight@resource://devtools/client/framework/toolbox.js:3539:11
highlightDomElement@resource://devtools/client/debugger/panel.js:157:17
highlightDomElement/<@resource://devtools/client/debugger/src/actions/toolbox.js:57:18
thunk/</</<@resource://devtools/client/debugger/src/actions/utils/middleware/thunk.js:29:45
bindActionCreator/<@resource://devtools/client/shared/vendor/redux.js:520:12
onDOMNodeMouseOver@resource://devtools/client/debugger/src/components/SecondaryPanes/Scopes.js:188:56
onMouseOver@resource://devtools/client/shared/components/reps/reps.js:7081:45

The Scopes pane can show small "highlight" icons next to DOM nodes. What seems to happen here is that a call to the highlighters is triggered at the end of the test, because the mouse cursor was over one of those "highlight icons". However, the test itself is not asserting anything related to mouse events.

Testing locally, I managed to get the same failure by leaving my mouse cursor exactly where one of those icons would show up, and running with --verify. Even then,it only failed pretty rarely.

Anyway, my guess is that either a previous test is moving the mouse in a wrong spot, or maybe this is the default position of the mouse on this platform. As far as fixing this, we could either:

• wait for requests to settle at the end of the test
• or move the mouse somewhere else at the beginning of the test
• update Toolbox::highlight to catch errors occurring after toolbox destroy

None of those fixes are really satisfactory as they just address the symptoms in various ways. The root cause is that we don't have a solution to handle async calls that are fired around the time of a toolbox destroy. I wish we had a way to cancel all devtools generated promises when the toolbox is destroyed.

Comment 22

[Julian Descottes [:jdescottes]]

Attachment: Bug 1645563 - Introduce devtools helper to swallow rejections under a condition

We very often have to introduce a pattern in our async methods in order to "swallow"
async errors if a panel was destroyed, if the toolbox was destroyed etc...

This is not an issue in a regular usage of the toolbox, but it can lead to many intermittent failures.
One solution could be to always wait for all requests to be completed before shutting down tests.
Another approach is to have an easy way of swallowing errors when a certain condition is true.

The helper added here will run the provide method in a try catch, and will only bubble up errors if the provided check fails.

Comment 23

[Julian Descottes [:jdescottes]]

Attachment: Bug 1645563 - Wrap Toolbox::highlight & unhighlight methods in safeAsyncMethod

Depends on D81675

This should fix the intermittents related to pending highlight requests when toolbox is destroyed.
See https://bugzilla.mozilla.org/show_bug.cgi?id=1645563#c21 for additional context

Comment 24

[Julian Descottes [:jdescottes]]

Attachment: Bug 1645563 - Enable browser_dbg-scopes-duplicated.js on windows platform

Depends on D81676

Comment 25

[Julian Descottes [:jdescottes]]

Attachment: Bug 1645563 - Wrap application panel updateWorkers method in safeAsyncMethod

Depends on D81677

We recently introduced a modification to updateWorkers in order to swallow errors after destroy.
Since I am suggesting to add a helper to take care of a similar issue in Toolbox::highlight I think we could also apply it here.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/59f5ba699cc7
Introduce devtools helper to swallow rejections under a condition r=nchevobbe
https://hg.mozilla.org/integration/autoland/rev/6c62392d88f7
Wrap Toolbox::highlight & unhighlight methods in safeAsyncMethod r=nchevobbe
https://hg.mozilla.org/integration/autoland/rev/948201d152d3
Enable browser_dbg-scopes-duplicated.js on windows platform r=nchevobbe
https://hg.mozilla.org/integration/autoland/rev/fc8c959faac2
Wrap application panel updateWorkers method in safeAsyncMethod r=ladybenko

Comment 27

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/59f5ba699cc7
https://hg.mozilla.org/mozilla-central/rev/6c62392d88f7
https://hg.mozilla.org/mozilla-central/rev/948201d152d3
https://hg.mozilla.org/mozilla-central/rev/fc8c959faac2

Comment 28

[Intermittent Failures Robot]

16 failures in 797 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 14
• mozilla-central: 2

Platform and build breakdown:

• windows10-64: 15
  • debug: 5
  • asan: 7
  • opt: 3
• windows10-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1645563&startday=2021-03-18&endday=2021-03-18&tree=all

Comment 29

[Julian Descottes [:jdescottes]]

The new failures are for a different error message, which is tracked by Bug 1699590 (A promise chain failed to handle a rejection: Actor 'SpecialPowers' destroyed before query 'Spawn' was resolved).

Comment 30

[Intermittent Failures Robot]

16 failures in 4257 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 14
• mozilla-central: 2

Platform and build breakdown:

• windows10-64: 15
  • debug: 5
  • asan: 7
  • opt: 3
• windows10-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1645563&startday=2021-03-15&endday=2021-03-21&tree=all