931044 - Write tests to ensure marionette doesn't leak disk space, memory and doesn't degrade in performance

Status: RESOLVED WONTFIX

(Remote Protocol :: Marionette, defect)

Description

[Malini Das [:mdas] - Away, not checking bugmail]

We want to make sure that Marionette won't keep unnecessary files or memory over time/sessions, and that we don't slow down over the course of a session or permanently across sessions.

Here are some ideas of what jgriffin, rwood and I would want to check:
* disk usage: ensure that imported files are deleted after sessions
How: we can prod the profile's tmp directory for this file

* performance: ensure no slowness occurs after:
** multiple OOP frame switches
** multiple navigations
** multiple element find operations
** mutliple execute_(async_)script calls
** multiple element .taps()
** multiple element .send_keys()
** all of the above after context and/or frame switches
** starting and deleting sessions repeatedly
How: One suggestion: Run N times, having an average over the first X runs, then calculate average of the last X runs and check if they're "wildly" different, for some definition of "wildly", something like 20% slowness increase detected maybe. Fail if so. 
** [jgriffin] We should ideally track memory growth over repeated runs as well, using about:memory reports.  I think performance is a good first metric, and I think that memory growth will take a bit of experimentation.  We may not be able to compare memory consumption before and after a test easily, since a lot of things can account for memory growth, but we could possibly compare memory growth between runs, ala areweslimyet.com.

* memory usage:
**  ensure that large memory stores are cleared after session is done (ie:  our dom cache should be GC'd), and after starting/deleting a few  sessions sequentially
** [jgriffin] I think Rob can help; we need to develop some mechanism of parsing/reporting the about:memory report for this case, and possibly for the endurance and MTBF tests as well
** [rwood] at the very least as a basic start we could just grab the b2g process RSS value (without needing the get_about_memory script) just to get up and running; before and after each test, and plot that over time (like the endurance tests currently do) and always expand later with about_memory dumps
How: I'm only familiar with  getting the about:memory dumps from the running device. I'm not sure how  to break that information down, or if there's a better way. I'd like to run this on desktop as well as on device

For the memory checking, I'd likely have to set up a local endurance CI instance where I can gather the about:memory dump artifacts, but for disk space/performance, I'd like to have those running in TBPL.

Comment 1

[Malini Das [:mdas] - Away, not checking bugmail]

tying into https://wiki.mozilla.org/Auto-tools/Goals/2013Q4#Top_Level, we should periodically run these for long periods of time (this goal states 3 days) to ensure we don't have problems with marionette over long periods of time.