Closed Bug 1565833 Opened 5 years ago Closed 5 years ago

Perma browser_aboutdebugging_devtoolstoolbox_reload.js | leaked 2 window(s) until shutdown when Gecko 70 merges to Beta on 2019-08-26

Categories

(DevTools :: about:debugging, defect)

Product:
DevTools
Component:
about:debugging
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox-esr60 unaffected, firefox-esr68 unaffected, firefox68 unaffected, firefox69 unaffected, firefox70+ verified)

Status:
VERIFIED FIXED
Milestone:
Firefox 70
Tracking Flags:
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- unaffected
firefox68 --- unaffected
firefox69 --- unaffected
firefox70 + verified

People

(Reporter: CosminS, Assigned: jdescottes)

References

Details

(Keywords: regression)

Attachments

(2 files)

Bug 1565833 - Wait for debugger-client requests to settle before closing the toolbox
47 bytes, text/x-phabricator-request
Details | Review
Bug 1565833 - Wait for requests to settle before reloading about:devtools-toolbox in tests
47 bytes, text/x-phabricator-request
Details | Review

[Tracking Requested - why for this release]:
Central as beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=256344087&revision=7c891a7092619e9f2bf3abb68b4542062420e0d4&searchStr=devtools&group_state=expanded

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=256344087&repo=try&lineNumber=7257

Even though the initial cause for these failures in tree (Bug 1354679) was backed out here https://hg.mozilla.org/mozilla-central/rev/cbac7fc3b4472c9f0f1a0d5c9be5c0f55f7419dd and was merged to central a few hrs later, these failures still happen on beta sims.
The tests are green on integration trees.
Our hunch is Bug 1560178 and the changes here: https://hg.mozilla.org/mozilla-central/rev/6071c24be566a2b3f2f98152456a540e9e893377
Gijs, Aryx could you please take a look over this and see why this happens? Thanks.

Flags: needinfo?(gijskruitbosch+bugs)

It looks like in https://bugzilla.mozilla.org/show_bug.cgi?id=1354679#c32 you backed out only the "tweak pause overlay" changeset, but bug 1544828 was filed immediately after 6363111ee8ce ("Automatically display the PausedDebuggerOverlay when the debugger is paused.") landed, and that doesn't look like it's been backed out?

Flags: needinfo?(csabou)

And e.g. https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=cbac7fc3b4472c9f0f1a0d5c9be5c0f55f7419dd&searchStr=devtools&selectedJob=256278810 and https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=cbac7fc3b4472c9f0f1a0d5c9be5c0f55f7419dd&searchStr=devtools&selectedJob=256273966 are after the backout and still orange. I don't know if the frequency has really gone down, it'll probably be hard-ish to tell until the weekend is over and normal push frequency resumes.

Fixed by backout of https://hg.mozilla.org/mozilla-central/rev/6363111ee8ce from bug 1354679.

Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(gijskruitbosch+bugs)
Flags: needinfo?(csabou)
Resolution: --- → FIXED

Hi, do you have an intermittent view of the failures?

https://treeherder.mozilla.org/intermittent-failures.html I'd like to see which platforms (or all) it was failing on and under what modes presumably debug...

Jason, this was filled as a beta-sim bug and is normally stared on tree as expected fail so no intermittent failures view.
But here's a failure rate from Bug 1544828 which is I think the same only for integration trees: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-07-08&endday=2019-07-15&tree=trunk&bug=1544828
It fails mostly on linux64 debug: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255650432&repo=autoland&lineNumber=7423 and sometimes on macosx1014-64 debug: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=256217833&repo=autoland&lineNumber=8097

This is still occurring on recent beta simulations:
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=pending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&revision=9bdb6707b69111f77ea9ad48f83f6f57d147b394&group_state=expanded&selectedJob=257522326

Aryx could the recent occurrences be from Bug 1497197?

Status: RESOLVED → REOPENED
Flags: needinfo?(aryx.bugmail)
Resolution: FIXED → ---

This looks to be from https://hg.mozilla.org/integration/autoland/rev/63e23c9ed252 with the changes from Bug 1354679 being repushed on tree. Also the culprit must be in this pushlog https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=dc32c9061a0a335dd582845703cd742c55be9c85&tochange=872c3c79f9060ac27b6b79435e4be869bb040e91 cause in yesterdays beta-sims the tests were green: https://tinyurl.com/yxhch8gj

Flags: needinfo?(jlaster)

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Keywords: regression

Hi, I'm a bit confused... I don't see any failures for this test on TRY or Central. I also can't reproduce this error locally in debug mode on osx. The test is extremely slow (requestLongerTimeout(5);) so it is very tough to get meaningful data...

https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=dt&fromchange=8f91a2e44c1aa6ba384f25d78a88b95e375dcb42&selectedJob=257892384

Do you have a good report where I can see the failures?

Flags: needinfo?(jlaster)

I had a try run last week with 50+ runs and no leaks, which seemed to confirm that the overlay was not to blame. comment

Here is a new try run on top of master with the overlay. I'm only testing the aboutdebugging directory because I want to have as many runs as possible:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5f16ad4345724a90e636a40e1e61032dadb8013b
https://treeherder.mozilla.org/#/jobs?repo=try&revision=38af0913f6097f5991568d814b1e58d5395fd90e

(In reply to Jason Laster [:jlast] from comment #12)

Hi, I'm a bit confused... I don't see any failures for this test on TRY or Central

As comment #0 points out, this is about what happens when central merges to beta. It doesn't look like your trypushes contain the relevant changes to test that configuration. You can probably import something like https://hg.mozilla.org/try/rev/7c891a7092619e9f2bf3abb68b4542062420e0d4 to do the same kind of tests.

It's likely something about the debugger or what is being debugged is nightly-only and so this test behaves differently in a beta configuration.

You can get the beta simulation by running the command from https://wiki.mozilla.org/Sheriffing/How_To/Beta_simulations#TRUNK_AS_EARLY_BETA but append --no-push to not run everything but hg commit the changes and run ./mach try chooser to pick the needed ones.

The test that is causing the issues was recently disabled for linux debug link. Do you think it is reasonable to disable for debug? The test is already using requestLongerTimeout(5); and is known for timeouts and OOMs, i'm not too surprised that it would leak in a failure.

Flags: needinfo?(jmaher)
Flags: needinfo?(jdescottes)
Flags: needinfo?(aryx.bugmail)

(In reply to Jason Laster [:jlast] from comment #16)

The test that is causing the issues was recently disabled for linux debug link.

Yes it was recently disabled on Linux debug. The perma fail for this test on debug mode started with Bug 1354679 (pause overlay). We should investigate why the debugger now leaks windows on reload.

Do you think it is reasonable to disable for debug?

We tried to do that in https://bugzilla.mozilla.org/show_bug.cgi?id=1544828, but it made the next test in the suite timeout (no idea why).
Either we skip this test and the next one in debug (potentially others? to confirm with a try push), or we fix the debugger leak that started with Bug 1354679.

Flags: needinfo?(jdescottes)

I also don't know if we really have an issue specific to the beta simulation or not here. There has been movement on both Bug 1354679 and Bug 1544828, so it's hard to say at the moment. I kicked a few try runs, we'll see how it goes:

Flags: needinfo?(jmaher)

I will take the bug for now.

Assignee: nobody → jdescottes
Status: REOPENED → ASSIGNED

Quick update to say that there is definitely something wrong linked with this test + beta simulation:

This avoids intermittent leaks in aboutdebugging tests relying on about:devtools-toolbox

Depends on D39174
Seems to fix the perma fail on beta

The patches attached here should fix the issue on beta osx debug:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=258088202&revision=d6d2e12be3c372c849181aa45c952ded5bd6db2d

The approach here is to wait for toolbox requests to be finished before reloading the toolbox and shutting down about:devtools-toolbox.
I still don't really know why the debugger patch regressed this (and I checked that the beta simulations were only perma-failing with Bug 1354679). My guess is that the debugger's initialization is not properly waiting for everything to be done before emitting the "ready" event and that should ideally be fixed. But for now that should be enough.

See Also: → 1568756
Pushed by jdescottes@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c314c3bab4f2
Wait for debugger-client requests to settle before closing the toolbox r=remote-debugging-reviewers,Ola,daisuke
https://hg.mozilla.org/integration/autoland/rev/6cd0d9de893d
Wait for requests to settle before reloading about:devtools-toolbox in tests r=remote-debugging-reviewers,Ola,daisuke
Flags: needinfo?(aryx.bugmail)

https://hg.mozilla.org/mozilla-central/rev/c314c3bab4f2
https://hg.mozilla.org/mozilla-central/rev/6cd0d9de893d

Status: ASSIGNED → RESOLVED
Closed: 5 years ago → 5 years ago
status-firefox70: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 70
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: