841211 - memory leak when running a test on repeat

Status: RESOLVED INVALID

(Firefox OS Graveyard :: General, defect)

Description

[Malini Das [:mdas] - Away, not checking bugmail]

Attachment: about:memory at startup

We've been seeing a lot of weird issues in our CI lately related to Bug 838786, and one of them looked like a memory issue since we've been seeing a lot of:

"E/memalloc(  109): /dev/pmem: No more pmem available"

in the logs, and usually right before our error.

I ran a single one of these tests on repeat to see if it reproduced our issue. It did, and I can see that the memory consumption increased after the suite was ran, despite killing the app it launches. If I run ./get_about_memory.py --minimize, a little of the memory is freed up.

The test (https://github.com/mozilla/gaia-ui-tests/blob/master/gaiatest/tests/test_wallpaper.py) kills all running apps, runs the settings app, sets a new wallpaper, sets it back to the default wallpaper, and quits. If I run it at most 70 times on a freshly booted phone, it will eventually hit a socket.error, and the memory consumption is really high.

I have the about:memory dumps from after startup, and from the point our test fails. I noticed that the b2g process' VSIZE grew by 50%, even after terminating the test and closing the settings app manually. 

It's not clear to me *where* the memory leak is. I see b2g's process' VSIZE is huge, and I see in the logs that the Main Process' memory at least doubles. Can anyone give some insight?

For STR, plug in your unagi, and, assuming you have python virtualenv installed on your machine:
git clone [e-mail]:mozilla/gaia-ui-tests.git
cd gaia-ui-tests
virtualenv venv
. venv/bin/activate
pip install marionette_client
python setup.py develop
adb forward tcp:2828 tcp:2828
gaiatest --address localhost:2828 --repeat=70 gaiatest/tests/test_wallpaper.py
deactivate

After this runs, check the about:memory usage.

Comment 1

[Malini Das [:mdas] - Away, not checking bugmail]

Attachment: about:memory after failure occurs and after killing settings app

Comment 2

[Malini Das [:mdas] - Away, not checking bugmail]

Attachment: about:memory after --minimize is run

Comment 3

[Alex Keybl [:akeybl]]

Memory leak after running an uncommon user scenario 70 times back to back, not a blocker.

Comment 4

[Malini Das [:mdas] - Away, not checking bugmail]

Hey jlebar, can we borrow your eyes to spot anything odd in the logs?

Comment 5

[Justin Lebar (not reading bugmail)]

RSS, which is the most meaningful measure of size, goes from 52.02mb to 56.56mb (after --minimize) in the main process.  The homescreen goes from 27.88mb to 21.29mb.

So neither of these seems to be the problem.

> "E/memalloc(  109): /dev/pmem: No more pmem available"

pmem isn't "real" memory; it's a type of graphics memory, I think.  Perhaps we're leaking that, and perhaps it's allocated in a way which doesn't show up here.

Comment 6

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Justin Lebar [:jlebar] from comment #5)
> RSS, which is the most meaningful measure of size, goes from 52.02mb to
> 56.56mb (after --minimize) in the main process.  The homescreen goes from
> 27.88mb to 21.29mb.
> 
> So neither of these seems to be the problem.
> 
> > "E/memalloc(  109): /dev/pmem: No more pmem available"
> 
> pmem isn't "real" memory; it's a type of graphics memory, I think.  Perhaps
> we're leaking that, and perhaps it's allocated in a way which doesn't show
> up here.

Ah, thanks. One thing I forgot to comment about was that when we hit the error point, the UI freezes, and it always freezes after we launch the app and the app is in view but with the dark overlay, ie: it freezesright before it transitions to being a usable app. You can't use anything on the screen, but, if you press the Home button, you will be able to access the Homescreen and use the phone again. Not sure if this helps with the issue.

Comment 7

[Malini Das [:mdas] - Away, not checking bugmail]

Important update to the STR:
Replace:

git clone [e-mail]:mozilla/gaia-ui-tests.git

with:

git clone git://github.com/malini/gaia-ui-tests-1.git gaia-ui-tests
git checkout remove_sp

The mozilla/gaia-ui-tests are using SpecialPowers which could add extra memory leakage. The branch I have in my github repo removes the usage of SpecialPowers. I generated the about memory logs using the tests I have in my repository.

Comment 8

[Nicholas Nethercote [inactive]]

Does this still happen?

Comment 9

[Malini Das [:mdas] - Away, not checking bugmail]

No, this is no longer reproducible.