Win debug mochitest-other runtime increased from ~90mins to 115mins in the last 2 weeks

RESOLVED FIXED in Firefox 18

Status

defect
--
major
RESOLVED FIXED
7 years ago
7 years ago

People

(Reporter: emorley, Assigned: bobbyholley)

Tracking

({regression})

Trunk
mozilla19
x86
Windows XP
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox18+ fixed)

Details

(Whiteboard: [qa-])

Attachments

(2 attachments, 3 obsolete attachments)

WinXp debug m-oth runs are now taking 115-120 mins, whereas two weeks ago they were more like 88-90 mins.
Last ~89 min WinXP run was:
https://tbpl.mozilla.org/?jobname=Rev3%20WINNT%205.1%20mozilla-central%20debug%20test%20mochitest-other&rev=710d9d21f533
-> https://tbpl.mozilla.org/php/getParsedLog.php?id=15241338&tree=Firefox

Then increased to ~95 mins in one of these pushes:
https://tbpl.mozilla.org/?jobname=mochitest-other&rev=fcd9acafa3ff
or
https://tbpl.mozilla.org/?jobname=mochitest-other&rev=f7c89de3ab43

Then increased to a whopping 112 mins in this push:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=mochitest-other&rev=633082d3e546

The first range might just be noise, but the latter seems excessive.

bholley, do you have any ideas? :-)
(Only seems to affect Windows, not
Blocks: 792036
Summary: Investigate WinXP debug mochitest-other runtime increase from ~90mins to 115mins in last 2 weeks → Investigate Win debug mochitest-other runtime increase from ~90mins to 115mins in last 2 weeks
Summary: Investigate Win debug mochitest-other runtime increase from ~90mins to 115mins in last 2 weeks → Win debug mochitest-other runtime increased from ~90mins to 115mins in the last 2 weeks
Seems pretty probably that it's related to the mega SpecialPowers testsuite fixup I pushed. I see two ways to narrow the problem down:

1 - Do some pushes and figure out which of those patches caused the regression. My money would be on dc84f6a237d7, a8583bee3c37, or 1f7c4fae49c0, so those are probably the only ones that we should check to start.

2 - Determine if one test (or a few tests) got a lot slower, or if the entire test suite just got slower across the board.

Ed, which do you think is easier? The first, I'd imagine?
I've just looked at the WinXP debug m-oth logs of your push and the one prior, and get:

Prior:
mochitest-chrome: 23 mins, 26 secs
mochitest-browser-chrome: 1 hrs, 1 mins, 58 secs
mochitest-ally: 5 mins, 35 secs
mochitest-ipcplugins: 1 mins, 14 secs

After the SpecialPowers fixup:
mochitest-chrome: 39 mins, 13 secs
mochitest-browser-chrome: 1 hrs, 42 secs
mochitest-ally: 5 mins, 40 secs
mochitest-ipcplugins: 1 mins, 8 secs

Will attach the test runtimes for mochitest-chrome.
Posted file mochitest-chrome original (obsolete) —
Posted file mochitest-chrome regressed (obsolete) —
Attachment #664904 - Attachment is obsolete: true
Attachment #664905 - Attachment is obsolete: true
Wow, awesome analysis Ed! Looks like the blame lies squarely with mochitest-chrome, which is lucky for our purposes, because most of the patches didn't touch mochitest-chrome at all. In fact, I'm pretty sure this was the only changeset that would have an affect on mochitest-chrome:

https://hg.mozilla.org/integration/mozilla-inbound/rev/1460d44ec8de
Investigating in a VM.
Assignee: nobody → bobbyholley+bmo
I've given up trying to reproduce this in a VM for the moment. Bisecting with some try pushes to test my theory about which commit we're dealing with:

https://tbpl.mozilla.org/?tree=Try&rev=00574bdb9666
https://tbpl.mozilla.org/?tree=Try&rev=2a2fd94ade0e
https://tbpl.mozilla.org/?tree=Try&rev=c2f895f02fed
Ok, so it looks like attaching the raw Components object is the culprit. That means either we're calling that function way too many times, or we're iterating over a gigantic number of scopes in FindInJSObjectScope. I added some logging that will tell us:

https://tbpl.mozilla.org/?tree=Try&rev=51f5b40ba655
Hm, looks like the issue _might_ be that we're iterating over a ton of scopes. I've fixed that over in my patches in bug 797821, so let's see if that fixes it:

https://tbpl.mozilla.org/?tree=Try&rev=6dfd704c8abb
Just wanted to say thank you for your persistence with this! :-)
Ahah! I think I regressed bug 722428.
Bingo. The hard edge here causes us to leak every window ever, a la bug 722428.

Kyle, did you ever figure out the pathology of this here? Why do we hold onto
the SpecialPowers object associated with the window after the window is gone?

It would be great if there was some way to test this...
Attachment #669730 - Flags: review?(khuey)
Sweet, now I have another hail-mary scapegoat for bug 792215!  Which leaks 20% of nsGlobalWindows, only on WinXP mochitest-chrome.  I'll see if that helps.
Attachment #669730 - Attachment is obsolete: true
Attachment #669730 - Flags: review?(khuey)
Attachment #669739 - Flags: review?(khuey)
Comment on attachment 669739 [details] [diff] [review]
Remove hard edge from SpecialPowers to window.Components. v2

Review of attachment 669739 [details] [diff] [review]:
-----------------------------------------------------------------

I would be curious to know why Windows is more affected than the other platforms.
Attachment #669739 - Flags: review?(khuey) → review+
Blocks: 792215
Blocks: 764713
https://hg.mozilla.org/mozilla-central/rev/3cb1f7ae777e
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla19
Can this be uplifted to aurora as FF18 is affected and this may help us building faster :) ?
Comment on attachment 669739 [details] [diff] [review]
Remove hard edge from SpecialPowers to window.Components. v2

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 792036
User impact if declined: Mochitest-chrome runs take 20 minutes longer on win dbg.
Testing completed (on m-c, etc.): m-c
Risk to taking this patch (and alternatives if risky): extremely low risk. test only. 
String or UUID changes made by this patch: none
Attachment #669739 - Flags: approval-mozilla-aurora?
Comment on attachment 669739 [details] [diff] [review]
Remove hard edge from SpecialPowers to window.Components. v2

Approving on aurora as the patch is low risk and check comment 24
Attachment #669739 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Whiteboard: [qa-]
You need to log in before you can comment on or make changes to this bug.