663416 - Better automated memory usage tests

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[Nicholas Nethercote [inactive]]

Talos Tp4 and Tp5 are primarily page load test, but they also do some memory usage measurements.  This is good, as it can detect some regressions, but there are definite shortcomings:

- They only open a single tab at a time.

- The metrics are curious, eg. "average a measurement that's taken every 20 seconds".

I see two options.

1. We could extend/improve Tp5 to be a better memory test.  

2. Or we could just create new tests;  Membuster (bug 631705) would be a good starting point.

Comment 1

[Nicholas Nethercote [inactive]]

Membuster is here, BTW:  http://random.pavlov.net/membuster/index.html

Comment 2

[Andrew McCreight [:mccr8]]

Would a high watermark be a useful measurement?

At a lower level, something measuring the number of page faults could be good, though I suppose those would implicitly show up in timing numbers, and I don't know how weird it would make comparing results between platforms.

Gregor et al's 2011 ISMM paper "Compartmental Memory Management in a Modern Web Browser" has some notions of measuring memory management in the browser that would be worth looking at.

Comment 3

[Nicholas Nethercote [inactive]]

I heard a rumour that Dave Hunt would be looking at this...

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I'm leading the Endurance Tests project, details of which can be found here: https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Endurance_Tests

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've put together an adhoc endurance test that recreates membuster. See bug 669613.

Comment 6

[Nicholas Nethercote [inactive]]

The endurance tests have come a long way, which is good.  But now we have Talos and endurance tests, both of which have their pros and cons, which I summarized here:

https://groups.google.com/forum/#!topic/mozilla.dev.platform/RIAOC8vys1s

I think improving Talos to do a better job is the right way forward.  

- Bug 695072 will (if implemented) allow multiple tabs to be used, and also allowing deep interactions with sites, and allow measurements to be taken at more interesting and appropriate points in time.

- Bug 685632 will (if implemented) allow "explicit" and "resident" (or something similar) to be measured in-browser, which means that the same metric can be used on all platforms.

It'll be fantastic if both those bugs can be completed -- we'll have a *much* better chance of detecting regressions.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

bug 685632 is under active development and should be deployed before too long.  

there are some requests for more useful pageloader actions.  Right now pageloader just does a loadURI() and waits for it to load.  The most basic change would be to have a list of links and click on the link to open a new tab.  When that tab is loaded close it.  This is something that might be a realistic replacement for most if not all pageloader talos tests.

Comment 8

[Nicholas Nethercote [inactive]]

Since this is a tracking bug and all the blocking bugs have "MemShrink" tags, there's no need for this to have a MemShrink tag.

Comment 9

[Nicholas Nethercote [inactive]]

With AWSY in place I think this bug can be closed.