807338 - Memory increase discovered by mozmill endurance tests - Mac 10.7&10.8 since Oct25

Status: RESOLVED WORKSFORME

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Mihaela Velimiroviciu (:mihaelav)]

Memory consumption increased since Oct 25th for Nightly builds on Mac 10.7 and 10.8.
Mac 10.7:
Explicit memory (MB): 	Maximum: 218 / Average: 102 -> Maximum: 409 / Average: 120
Resident memory (MB): 	Maximum: 663 / Average: 203 -> Maximum: 852 / Average: 226

Mac 10.8:
Explicit memory (MB): 	Maximum: 368 / Average: 117 -> Maximum: 460 / Average: 128
Resident memory (MB): 	Maximum: 791 / Average: 221 -> Maximum: 912 / Average: 234

Reports:
* 24 Oct: 
    - http://mozmill-ci.blargon7.com/#/endurance/report/2e7486a8c08b5541b9074117e0c81148
    - http://mozmill-ci.blargon7.com/#/endurance/report/2e7486a8c08b5541b9074117e0c7f2e6
* 25 Oct:
    - http://mozmill-ci.blargon7.com/#/endurance/report/2e7486a8c08b5541b9074117e0f01342
    http://mozmill-ci.blargon7.com/#/endurance/report/2e7486a8c08b5541b9074117e0eff3ec

I was able to reproduce it locally on Mac 10.7:
Explicit memory (MB):	Maximum: 393 / Average: 129 -> Maximum: 458 / Average: 137
Resident memory (MB):	Maximum: 615 / Average: 218 -> Maximum: 676 / Average: 230
Reports:
* 24 Oct: http://mozmill-crowd.blargon7.com/#/endurance/report/190d57cf64e3c5c3996c0c1059435b2b
* 25 Oct: http://mozmill-crowd.blargon7.com/#/endurance/report/190d57cf64e3c5c3996c0c10594552d1

Pushlog: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=93cc1ee94291&tochange=5c82f5a5e90d

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's a massive jump if it takes twice that much memory now. So please use tinderbox builds to nail down this problem ASAP. I will put this as a P1 item for the upcoming week.

Comment 2

[Andrew McCreight [:mccr8]]

This doesn't show up on the endurance tests for any other platforms? Nothing in the range jumps out at me as being something that might cause memory problems.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Henrik: The increase is not twice as much, I suspect you are comparing explicit to resident. The important numbers here are the average values:

Mac 10.7:
Average explicit memory (MB): 102 -> 120
Average resident memory (MB): 203 -> 226

Mac 10.8:
Average explicit memory (MB): 117 -> 128
Average resident memory (MB): 221 -> 234

It also appears that the increase is large due to the testOpenAndCloseMultipleWindows test, which we're already aware is not giving realistic results. This will be addressed by bug 788531.

Mihaela: Can you try to replicate this with a delay of 5 seconds?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Pulling off from our automation sprint given that Mihaela is investigating this issue.

Comment 5

[Mihaela Velimiroviciu (:mihaelav)]

(In reply to Dave Hunt (:davehunt) from comment #3)
> Mihaela: Can you try to replicate this with a delay of 5 seconds?

I used a delay of 5 seconds when running the tests and it seems that now there is no difference in the used memory between Oct 24th and 25th  Nightlies.

Reports:
* Oct 24: http://mozmill-crowd.blargon7.com/#/endurance/report/190d57cf64e3c5c3996c0c10599e0bda
Explicit memory (MB):	Minimum: 65 / Maximum: 166 / Average: 87
Resident memory (MB):	Minimum: 121 / Maximum: 294 / Average: 161
* Oct 25: http://mozmill-crowd.blargon7.com/#/endurance/report/190d57cf64e3c5c3996c0c10599fdfdf
Explicit memory (MB):	Minimum: 65 / Maximum: 167 / Average: 87
Resident memory (MB):	Minimum: 121 / Maximum: 294 / Average: 161

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Mihaela Velimiroviciu [QA] (:mihaela) from comment #5)
> I used a delay of 5 seconds when running the tests and it seems that now
> there is no difference in the used memory between Oct 24th and 25th 
> Nightlies.

That's great, thanks Mihaela!