Closed Bug 764823 Opened 12 years ago Closed 12 years ago

Memory spike discovered by endurance tests on Nightly Linux and Mac between June 12 and 13

Categories

(Mozilla QA Graveyard :: Mozmill Tests, defect)

All
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: mihaelav, Assigned: davehunt)

Details

(Whiteboard: [mozmill-endurance])

There is a memory increase in Nightly endurance for Linux and Mac.
The test that use more memory are:
/testTabbedBrowsing_OpenNewTab/test1.js	-> testOpenNewTab
/testTabView_OpenNewTab/test1.js -> testOpenNewTab
/testBookmarks_OpenAllInTabs/test1.js -> testOpenAllBookmarksInTabs

Ubuntu 32 bit:
Explicit memory (MB): Maximum: 115 / Average: 71 -> Maximum: 148 / Average: 76
Resident memory (MB): Maximum: 140 / Average: 101 -> Maximum: 173 / Average: 105
Reports: 
- http://mozmill-ci.blargon7.com/#/endurance/report/ee284dfa9d664754a65676abbc0e27a4 (June 12)
- http://mozmill-ci.blargon7.com/#/endurance/report/e67171ea696231bb192f565615033340 (June 13)

Ubuntu 64 bit:
Explicit memory (MB): Maximum: 133 / Average: 86  -> Maximum: 186 / Average: 103
Resident memory (MB): Maximum: 159 / Average: 121 -> Maximum: 218 / Average: 137
Reports:
- http://mozmill-ci.blargon7.com/#/endurance/report/ee284dfa9d664754a65676abbc0fc496 (June 12)
- http://mozmill-ci.blargon7.com/#/endurance/report/e67171ea696231bb192f56561503415e (June 13)

Mac 10.6.8:
Explicit memory (MB): Maximum: 145 / Average: 91 -> Maximum: 160 / Average: 100
Resident memory (MB): Maximum: 165 / Average: 122 -> Maximum: 184 / Average: 132
Reports:
- http://mozmill-ci.blargon7.com/#/endurance/report/ee284dfa9d664754a65676abbc0f15a7 (june 12)
- http://mozmill-ci.blargon7.com/#/endurance/report/e67171ea696231bb192f56561503a63d (June 13)

Mac 10.7.4:
Explicit memory (MB): Maximum: 128 / Average: 90 -> Maximum: 180 / Average: 104
Resident memory (MB): Maximum: 221 / Average: 172 -> Maximum: 273 / Average: 186
Reports:
- http://mozmill-ci.blargon7.com/#/endurance/report/ee284dfa9d664754a65676abbc0f3ff2 (june 12)
- http://mozmill-ci.blargon7.com/#/endurance/report/e67171ea696231bb192f56561503d0b4 (June 13)
I can attempt to replicate this on an Ubuntu VM, and if successful will bisect to find the change that caused the increase. I don't currently have a Linux build set up though (now is as good a time as any), so if anyone else is better placed to investigate this please go ahead.
Assignee: nobody → dave.hunt
Status: NEW → ASSIGNED
Whiteboard: [mozmill-endurance]
I can reproduce this issue locally on an Ubuntu 12.04 64-bit machine and will go on with trying to find out the cause for this increase.

BuildID: 20120612030527
http://mozmill-crowd.blargon7.com/#/endurance/report/29fc09ba0c8360d637617903a01eef02
Explicit memory (MB):	Minimum: 73 / Maximum: 207 / Average: 108
Resident memory (MB):	Minimum: 103 / Maximum: 232 / Average: 137

BuildID: 20120613030535
http://mozmill-crowd.blargon7.com/#/endurance/report/29fc09ba0c8360d637617903a01f51a4
Explicit memory (MB):	Minimum: 74 / Maximum: 212 / Average: 119
Resident memory (MB):	Minimum: 104 / Maximum: 242 / Average: 148
Looks good. Any progress with reducing the regression range?
As I said in comment 3, I started investigating this issue a while ago. I had to stop since something happened on my machine that messes with endurance results. I don't know what happened, but I always get results at least 20MB higher that they should be, although I am running the same tests on the same builds.

Before this mess happened, I got to the following results:
Changeset: a78601d88586
Explicit memory (MB): Minimum: 73 / Maximum: 219 / Average: 113
Resident memory (MB): Minimum: 116 / Maximum: 258 / Average: 154

Changeset: 62d7c3276ceb
Explicit memory (MB): Minimum: 72 / Maximum: 194 / Average: 107
Resident memory (MB): Minimum: 116 / Maximum: 231 / Average: 148

The culprit must be between these two changesets. Considering that most of the changesets between them have been backed out in the same pushlog, only three remain:
4c34e1a4cbb5 - Mark Capella — Bug 760881 - decomtaminate Select Row / Column() on accessible tables, r=tbsaunde
828da55601e7 - Bill McCloskey — Bug 761739 - Make timer-triggered GCs be incremental (r=smaug)
3ecdf0d8b4fa - Bill McCloskey — Bug 758034 - Create separate inter-slice GC timer (r=smaug)

I suspect 828da55601e7 (Bug 761739 - Make timer-triggered GCs be incremental).
I would agree that enabling incremental GCs is a likely cause for this memory increase. I'm going to close this issue as INCOMPLETE because the memory levels in nightly builds dropped back to levels shown prior to this increase. The tinderbox builds are no longer available to help us replicate this on Mozmill CI, but now that we have this ability we should use it whenever we see similar memory regressions in the future.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → INCOMPLETE
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.