We want to make sure that Marionette won't keep unnecessary files or memory over time/sessions, and that we don't slow down over the course of a session or permanently across sessions. Here are some ideas of what jgriffin, rwood and I would want to check: * disk usage: ensure that imported files are deleted after sessions How: we can prod the profile's tmp directory for this file * performance: ensure no slowness occurs after: ** multiple OOP frame switches ** multiple navigations ** multiple element find operations ** mutliple execute_(async_)script calls ** multiple element .taps() ** multiple element .send_keys() ** all of the above after context and/or frame switches ** starting and deleting sessions repeatedly How: One suggestion: Run N times, having an average over the first X runs, then calculate average of the last X runs and check if they're "wildly" different, for some definition of "wildly", something like 20% slowness increase detected maybe. Fail if so. ** [jgriffin] We should ideally track memory growth over repeated runs as well, using about:memory reports. I think performance is a good first metric, and I think that memory growth will take a bit of experimentation. We may not be able to compare memory consumption before and after a test easily, since a lot of things can account for memory growth, but we could possibly compare memory growth between runs, ala areweslimyet.com. * memory usage: ** ensure that large memory stores are cleared after session is done (ie: our dom cache should be GC'd), and after starting/deleting a few sessions sequentially ** [jgriffin] I think Rob can help; we need to develop some mechanism of parsing/reporting the about:memory report for this case, and possibly for the endurance and MTBF tests as well ** [rwood] at the very least as a basic start we could just grab the b2g process RSS value (without needing the get_about_memory script) just to get up and running; before and after each test, and plot that over time (like the endurance tests currently do) and always expand later with about_memory dumps How: I'm only familiar with getting the about:memory dumps from the running device. I'm not sure how to break that information down, or if there's a better way. I'd like to run this on desktop as well as on device For the memory checking, I'd likely have to set up a local endurance CI instance where I can gather the about:memory dump artifacts, but for disk space/performance, I'd like to have those running in TBPL.
Assignee: nobody → mdas
tying into https://wiki.mozilla.org/Auto-tools/Goals/2013Q4#Top_Level, we should periodically run these for long periods of time (this goal states 3 days) to ensure we don't have problems with marionette over long periods of time.
Status: NEW → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.