Closed Bug 1482797 Opened Last year Closed Last year
.47 - 11 .13% Resident Memory (android-em-4-2-x86, android-em-4-3-armv7-api16) regression on push 2b2bd723ebc8 (Sat Aug 11 2018)
We have detected an awsy regression from push: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=32ad5fef292ac00aaf5049c4ceceb5d93f173809&tochange=2b2bd723ebc8f15f747c4783e8f6d62f3e6d0d0a As author of one of the patches included in that push, we need your help to address this regression. Regressions: 11% Resident Memory android-em-4-3-armv7-api16 opt 142,989,143.04 -> 158,907,834.83 8% Resident Memory android-em-4-2-x86 opt 150,113,381.12 -> 162,824,984.50 You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=14900 On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format. To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests
Product: Testing → Firefox Build System
Eric, where are those android awsy tests? I don't find them on treeherder (meaning I can't find more detailed data to make sense of the regression).
Flags: needinfo?(mh+mozilla) → needinfo?(erahm)
(In reply to Mike Hommey [:glandium] from comment #1) > Eric, where are those android awsy tests? I don't find them on treeherder > (meaning I can't find more detailed data to make sense of the regression). It's a separate test called 'test_awsy_lite.html'  (not to be confused with 'awsy-lite'). I'm guessing it gets run via |./mach test mobile/android/tests/browser/chrome/test_awsy_lite.html| cc'd Geoff who wrote the test.  https://searchfox.org/mozilla-central/source/mobile/android/tests/browser/chrome/test_awsy_lite.html
So it's a chrome browser test. I found it in M(c4)... but we don't have about:memory snapshots for it like the normal awsy, apparently :( How do we get more information?
The test logs some raw values: https://treeherder.mozilla.org/logviewer.html#?job_id=193503661&repo=mozilla-inbound&lineNumber=1882 [task 2018-08-12T09:37:37.659Z] 09:37:37 INFO - 23 INFO TEST-START | mobile/android/tests/browser/chrome/test_awsy_lite.html [task 2018-08-12T09:37:37.659Z] 09:37:37 INFO - 24 INFO Fresh start | Resident Memory: 133861376 [task 2018-08-12T09:38:10.331Z] 09:38:10 INFO - 25 INFO Fresh start [+30s] | Resident Memory: 131563520 [task 2018-08-12T09:38:10.331Z] 09:38:10 INFO - 26 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/baidu.com/www.baidu.com/s@wd=mozilla.html] [task 2018-08-12T09:38:20.659Z] 09:38:20 INFO - 27 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/twitter.com/twitter.com/ICHCheezburger.html] [task 2018-08-12T09:38:42.017Z] 09:38:42 INFO - 28 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/msn.com/www.msn.com/index.html] [task 2018-08-12T09:39:03.975Z] 09:39:03 INFO - 29 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/163.com/www.163.com/index.html] [task 2018-08-12T09:40:11.453Z] 09:40:11 INFO - 30 INFO opening tab with url [http://mochi.test:8888/chrome/mobile/android/tests/browser/chrome/tp5/bbc.co.uk/www.bbc.co.uk/news/index.html] [task 2018-08-12T09:40:44.136Z] 09:40:44 INFO - 31 INFO After tabs | Resident Memory: 184250368 [task 2018-08-12T09:41:15.908Z] 09:41:15 INFO - 32 INFO After tabs [+30s] | Resident Memory: 187809792 [task 2018-08-12T09:41:15.909Z] 09:41:15 INFO - 33 INFO After tabs [+30s, forced GC] | Resident Memory: 187809792 [task 2018-08-12T09:41:15.909Z] 09:41:15 INFO - 34 INFO Tabs closed | Resident Memory: 174030848 [task 2018-08-12T09:41:47.875Z] 09:41:47 INFO - 35 INFO Tabs closed [+30s] | Resident Memory: 167731200 [task 2018-08-12T09:41:47.876Z] 09:41:47 INFO - 36 INFO Tabs closed [+30s, forced GC] | Resident Memory: 167731200 but, as far as I remember, that's all the info the test produces. You can run it locally to debug: ./mach mochitest mobile/android/tests/browser/chrome/test_awsy_lite.html from an Android build context should work fine.
So this turns out to be due to the fact the new NDK defines MADV_FREE, which mozjemalloc then happily uses instead of MADV_DONTNEED, and which has different properties: - it's not supported on old kernels, so practically speaking it does nothing on the test emulator, so mozjemalloc thinks it freed dirty pages to the OS, but actually didn't. So in the end, even after I was able to get about:memory dumps from the test, they weren't showing the regression in any useful way (they only appeared as a difference in outside-brk anonymous memory, which is not specific enough to diagnostic anything). - when it's supported, it lazily releases memory to the OS, so the memory still counts in Resident memory, until there is memory pressure that makes the kernel actually release it. I filed bug 1406304 a while ago for actual support for MADV_FREE, and come to think of it, I should have forced it to be unused even when defined until it's supported properly back then. It doesn't matter for the mozilla.org-produced builds, because they're built with old-enough kernel/libc headers, but it does matter for downstream. So this needs to be fixed now for our android builds, which end up using it, and I think this should be uplifted too.
Component: General → Memory Allocator
Product: Firefox Build System → Core
Comment on attachment 9000182 [details] [diff] [review] Don't use MADV_FREE on Linux until we support it properly [Approval Request Comment] User impact if declined: More memory usage than expected on Linux, for (some) downstream builds. Risk to taking this patch (and alternatives if risky): The patch is a no-op for Linux builds produced by Mozilla, and for Android builds before bug 1482330. It reduces memory usage for some downstream Linux builds. String or UUID changes made by this patch: N/A
Attachment #9000182 - Flags: review?(n.nethercote) → review+
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/0d27c68cfaf8 Don't use MADV_FREE on Linux until we support it properly. r=njn
Comment on attachment 9000182 [details] [diff] [review] Don't use MADV_FREE on Linux until we support it properly No-op patch for Mozilla-built Firefox builds, but makes things better for downstream distros. Approved for 62.0b18 and ESR 60.2.
we see an improvement: == Change summary for alert #15042 (as of Wed, 15 Aug 2018 16:41:55 GMT) == Improvements: 13% Resident Memory android-em-4-3-armv7-api16 opt 163,495,678.00 -> 142,318,760.75 For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=15042
You need to log in before you can comment on or make changes to this bug.