Closed Bug 931044 Opened 7 years ago Closed 3 years ago
Write tests to ensure marionette doesn't leak disk space, memory and doesn't degrade in performance
We want to make sure that Marionette won't keep unnecessary files or memory over time/sessions, and that we don't slow down over the course of a session or permanently across sessions. Here are some ideas of what jgriffin, rwood and I would want to check: * disk usage: ensure that imported files are deleted after sessions How: we can prod the profile's tmp directory for this file * performance: ensure no slowness occurs after: ** multiple OOP frame switches ** multiple navigations ** multiple element find operations ** mutliple execute_(async_)script calls ** multiple element .taps() ** multiple element .send_keys() ** all of the above after context and/or frame switches ** starting and deleting sessions repeatedly How: One suggestion: Run N times, having an average over the first X runs, then calculate average of the last X runs and check if they're "wildly" different, for some definition of "wildly", something like 20% slowness increase detected maybe. Fail if so. ** [jgriffin] We should ideally track memory growth over repeated runs as well, using about:memory reports. I think performance is a good first metric, and I think that memory growth will take a bit of experimentation. We may not be able to compare memory consumption before and after a test easily, since a lot of things can account for memory growth, but we could possibly compare memory growth between runs, ala areweslimyet.com. * memory usage: ** ensure that large memory stores are cleared after session is done (ie: our dom cache should be GC'd), and after starting/deleting a few sessions sequentially ** [jgriffin] I think Rob can help; we need to develop some mechanism of parsing/reporting the about:memory report for this case, and possibly for the endurance and MTBF tests as well ** [rwood] at the very least as a basic start we could just grab the b2g process RSS value (without needing the get_about_memory script) just to get up and running; before and after each test, and plot that over time (like the endurance tests currently do) and always expand later with about_memory dumps How: I'm only familiar with getting the about:memory dumps from the running device. I'm not sure how to break that information down, or if there's a better way. I'd like to run this on desktop as well as on device For the memory checking, I'd likely have to set up a local endurance CI instance where I can gather the about:memory dump artifacts, but for disk space/performance, I'd like to have those running in TBPL.
Assignee: nobody → mdas
tying into https://wiki.mozilla.org/Auto-tools/Goals/2013Q4#Top_Level, we should periodically run these for long periods of time (this goal states 3 days) to ensure we don't have problems with marionette over long periods of time.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.