1482797 - 8.47 - 11.13% Resident Memory (android-em-4-2-x86, android-em-4-3-armv7-api16) regression on push 2b2bd723ebc8 (Sat Aug 11 2018)

Status: RESOLVED FIXED

(Core :: Memory Allocator, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected an awsy regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=32ad5fef292ac00aaf5049c4ceceb5d93f173809&tochange=2b2bd723ebc8f15f747c4783e8f6d62f3e6d0d0a

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 11%  Resident Memory android-em-4-3-armv7-api16 opt      142,989,143.04 -> 158,907,834.83
  8%  Resident Memory android-em-4-2-x86 opt              150,113,381.12 -> 162,824,984.50


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=14900

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

Comment 1

[Mike Hommey [:glandium]]

Eric, where are those android awsy tests? I don't find them on treeherder (meaning I can't find more detailed data to make sense of the regression).

Comment 2

[Eric Rahm [:erahm]]

(In reply to Mike Hommey [:glandium] from comment #1)
> Eric, where are those android awsy tests? I don't find them on treeherder
> (meaning I can't find more detailed data to make sense of the regression).

It's a separate test called 'test_awsy_lite.html' [1] (not to be confused with 'awsy-lite'). I'm guessing it gets run via |./mach test mobile/android/tests/browser/chrome/test_awsy_lite.html|

cc'd Geoff who wrote the test.

[1] https://searchfox.org/mozilla-central/source/mobile/android/tests/browser/chrome/test_awsy_lite.html

Comment 3

[Mike Hommey [:glandium]]

So it's a chrome browser test. I found it in M(c4)... but we don't have about:memory snapshots for it like the normal awsy, apparently :(
How do we get more information?

Comment 4

[Geoff Brown [:gbrown]]

The test logs some raw values:

https://treeherder.mozilla.org/logviewer.html#?job_id=193503661&repo=mozilla-inbound&lineNumber=1882

[task 2018-08-12T09:37:37.659Z] 09:37:37     INFO -  23 INFO TEST-START | mobile/android/tests/browser/chrome/test_awsy_lite.html
[task 2018-08-12T09:37:37.659Z] 09:37:37     INFO -  24 INFO Fresh start | Resident Memory: 133861376
[task 2018-08-12T09:38:10.331Z] 09:38:10     INFO -  25 INFO Fresh start [+30s] | Resident Memory: 131563520
[task 2018-08-12T09:38:10.331Z] 09:38:10     INFO -  26 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/baidu.com/www.baidu.com/s@wd=mozilla.html]
[task 2018-08-12T09:38:20.659Z] 09:38:20     INFO -  27 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/twitter.com/twitter.com/ICHCheezburger.html]
[task 2018-08-12T09:38:42.017Z] 09:38:42     INFO -  28 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/msn.com/www.msn.com/index.html]
[task 2018-08-12T09:39:03.975Z] 09:39:03     INFO -  29 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/163.com/www.163.com/index.html]
[task 2018-08-12T09:40:11.453Z] 09:40:11     INFO -  30 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/bbc.co.uk/www.bbc.co.uk/news/index.html]
[task 2018-08-12T09:40:44.136Z] 09:40:44     INFO -  31 INFO After tabs | Resident Memory: 184250368
[task 2018-08-12T09:41:15.908Z] 09:41:15     INFO -  32 INFO After tabs [+30s] | Resident Memory: 187809792
[task 2018-08-12T09:41:15.909Z] 09:41:15     INFO -  33 INFO After tabs [+30s, forced GC] | Resident Memory: 187809792
[task 2018-08-12T09:41:15.909Z] 09:41:15     INFO -  34 INFO Tabs closed | Resident Memory: 174030848
[task 2018-08-12T09:41:47.875Z] 09:41:47     INFO -  35 INFO Tabs closed [+30s] | Resident Memory: 167731200
[task 2018-08-12T09:41:47.876Z] 09:41:47     INFO -  36 INFO Tabs closed [+30s, forced GC] | Resident Memory: 167731200

but, as far as I remember, that's all the info the test produces.


You can run it locally to debug: ./mach mochitest mobile/android/tests/browser/chrome/test_awsy_lite.html from an Android build context should work fine.

Comment 5

[Mike Hommey [:glandium]]

So this turns out to be due to the fact the new NDK defines MADV_FREE, which mozjemalloc then happily uses instead of MADV_DONTNEED, and which has different properties:
- it's not supported on old kernels, so practically speaking it does nothing on the test emulator, so mozjemalloc thinks it freed dirty pages to the OS, but actually didn't. So in the end, even after I was able to get about:memory dumps from the test, they weren't showing the regression in any useful way (they only appeared as a difference in outside-brk anonymous memory, which is not specific enough to diagnostic anything).
- when it's supported, it lazily releases memory to the OS, so the memory still counts in Resident memory, until there is memory pressure that makes the kernel actually release it.

I filed bug 1406304 a while ago for actual support for MADV_FREE, and come to think of it, I should have forced it to be unused even when defined until it's supported properly back then. It doesn't matter for the mozilla.org-produced builds, because they're built with old-enough kernel/libc headers, but it does matter for downstream.

So this needs to be fixed now for our android builds, which end up using it, and I think this should be uplifted too.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Don't use MADV_FREE on Linux until we support it properly

Comment 7

[Mike Hommey [:glandium]]

Comment on attachment 9000182 [details] [diff] [review]
Don't use MADV_FREE on Linux until we support it properly

[Approval Request Comment]
User impact if declined: More memory usage than expected on Linux, for (some) downstream builds.
Risk to taking this patch (and alternatives if risky): The patch is a no-op for Linux builds produced by Mozilla, and for Android builds before bug 1482330. It reduces memory usage for some downstream Linux builds.
String or UUID changes made by this patch: N/A

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0d27c68cfaf8
Don't use MADV_FREE on Linux until we support it properly. r=njn

Comment 9

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/0d27c68cfaf8

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9000182 [details] [diff] [review]
Don't use MADV_FREE on Linux until we support it properly

No-op patch for Mozilla-built Firefox builds, but makes things better for downstream distros. Approved for 62.0b18 and ESR 60.2.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/8ad88c2fad5c

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/0828425992f2

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

we see an improvement:
== Change summary for alert #15042 (as of Wed, 15 Aug 2018 16:41:55 GMT) ==

Improvements:

 13%  Resident Memory android-em-4-3-armv7-api16 opt      163,495,678.00 -> 142,318,760.75

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=15042