792165 - Add a test/benchmark for measuring memory usage

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

We should add a talos test to measure how much memory we use (both as a baseline without any content loaded and with a few tabs) that we can use as a metric for reducing memory usage.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

I noticed https://areweslimyet.com/ hanging off the MemShrink wiki page - it would be good to get fennec builds on that as well. Looking at that also makes me think that perhaps there are better benchmarks than talos tests, so I'm making this more general to cover whatever benchmark is most suitable.

Comment 2

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Be aware that there is tp4m_main_rss_nochrome -- http://graphs.mozilla.org/graph.html#tests=[[136,11,20]]&sel=none&displayrange=7&datatype=running. I don't know much about it; it seems quite simplistic...might be the place to start.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

I did some quick comparisons between a 512-meg ARMv6 device and a similarly-specced emulator image and I do get sizable differences (~70 megs in vsize) in the about:memory stats between the two. Comparing the two outputs indicates that most of the disparity comes from a handful of dalvik memory blocks such as:

size/other-files/dalvik-heap [---p]
size/other-files/dalvik-mark-stack [rw-p]

I assume that this is because the emulator is running Android 4.1 while the device I have is running Android 2.3.4 and there are probably some dalvik changes between the two. Since the emulator won't emulate Firefox on Gingerbread-level AVDs, and since most ARMv6 devices will be running older Android versions, I don't think it makes sense to try and set up a benchmark using the android emulator. Instead I'll focus on getting it running on a real device.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kartikaya Gupta (:kats) from comment #3)
> Since the emulator won't emulate Firefox on Gingerbread-level AVDs

To clarify, this is because of GPU emulation only being supported on 4.0.3 and higher level AVDs. I tried running an ARMv6 APK off a Gingerbread emulator without GPU emulation and it crashed on startup.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Just an update: I almost have this set up now. The setup is as close to AWSY as I could make it. There's an addon that runs within Fennec and runs through the same test that AWSY does (i.e. grab memory dump, wait 30 seconds, grab another memory dump, open a bunch of tabs, grab another memory dump, ...).

The main problem I was having was to be able to host the tp5 pageset somewhere that Fennec could use it without having the web server interfere with Fennec. The pages also need to be hosted on separate origins which complicated things (currently AWSY does it by using localhost:8001 through localhost:8100). I ended up writing a tcp stream muxer/demuxer and use adb's port forwarding mechanism to get this done. All outgoing connections from Fennec are intercepted, muxed into a single tcp connection that is forwarded over USB to the host machine, where the demuxer unpacks the connections and sends them out the nginx web server. The responses follow the same path in reverse. This mostly works except I'm running into a few hiccups when connections are closed, they don't get torn down properly and it ends up in a bad state.

Once that is fixed I'll set up a cron job to fetch the latest inbound code, build it, throw the build onto the device, reset the profile to a clean-start profile with the addon (that I have saved in a zipfile), set up the port forwarding, and start fennec to run the test. The memory data gets dumped to logcat which I can scrape and then upload to AWSY.

Current code is at https://github.com/staktrace/awsy-armv6

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Update: I now have a working setup to download and run the benchmark on inbound builds. Code and README at https://github.com/staktrace/awsy-armv6 has been updated to latest.

Now I just need to talk to :johns about how to format the data and get it uploaded to AWSY (or alternatively set up a new website to visualize the data, but that seems unnecessary).

Note that the benchmark loads 17 sites instead of the 100 from tp5 because on the device I'm running the benchmark on that's how many it can load before it OOMs. Also I had to exclude the wikipedia page because it has javascript that redirects to the mobile site, and that causes the benchmark to hang because non-localhost URL accesses are blocked. The URLs being loaded can be seen at https://github.com/staktrace/awsy-armv6/blob/master/fennec-addon-awsy/bootstrap.js#L11

Comment 7

[Kartikaya Gupta (email:[e-mail])]

It seems like even loading 17 sites might be too much; a couple of times now the device has locked up hard and I needed to reboot it. I might have to reduce it to 15 sites.

Comment 8

[Kartikaya Gupta (email:[e-mail])]

:johns, ping. My setup was broken for a bit but now it's working again, and I'm still interested in getting the data uploaded somewhere and visible publicly so we can easily watch for regressions.

Comment 9

[Kartikaya Gupta (email:[e-mail])]

I guess I dropped the ball on this. I ran my setup again and it looks like there's been some regressions since I last ran it a while ago. Maybe I'll just fork AWSY and spin up an areweslimyet.mobi with the data for Fennec. At least it'll force me to keep gathering the data and help us catch subsequent regressions. Still would be nice to just hook up the data to AWSY and have it all in one spot.

Comment 10

[John Schoenick [:johns]]

We discussed how to get this going on AWSY on IRC, clearing needinfo for now

Comment 11

[Kartikaya Gupta (email:[e-mail])]

http://people.mozilla.com/~kgupta/awsy/

Comment 12

[Nicholas Nethercote [inactive]]

Cool.

Comment 13

[Kartikaya Gupta (email:[e-mail])]

Data is now integrated into:

https://www.areweslimyet.com/mobile/