1258468 - Consider Nimbledroid for performance analysis

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, defect)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I discovered Nimbledroid @ Droidcon this past week. You upload your APK (they have an API) and they:
* Print metrics about the APK like method count & APK size
* Create automated interaction scenarios in your app (i.e. click through stuff using heuristics) and measure duration (seconds), memory usage, network access, and disk IO
  - Break down which methods may be causing issues
* Compare your app against the top-rated apps (e.g. there is a warning when startup time exceeds 2 seconds (on which device?), which is the time the top 100 apps average)
* Track the above changes across builds

They also do analyses on the performance characteristics of top apps so it could be worth following their blog [1].

I spoke with them on the floor and they mentioned:
* if there's something that doesn't work for us, let them know and they'll do their best
* They have a (not yet published) rest API to download the test data (e.g. they don't have notifications for when your app "regresses" – they're trying to hone the metric – but we could make this ourselves from the test data)

Geoff, I think you have the most context on our perf analysis tools – what are your thoughts? Do you think this is worth including as part of our metrics? Could it replace anything we're maintaining ourselves?

[1]: http://blog.nimbledroid.com/

Comment 1

[Geoff Brown [:gbrown]]

Nimbledroid looks interesting - thanks for pointing it out.

We track APK size and some memory use on Perfherder, so there's certainly some overlap there.

I imagine it would be easy to modify our build jobs to upload builds to Nimbledroid. Would we try to get results back from Nimbledroid and into Perfherder, or would we just track changes and view results on Nimbledroid? If using Nimbledroid, I wonder if that would decrease the "visibility" of results: If you are a platform developer who doesn't normally work with the mobile team but is wondering how a recent change affected performance on Android, how would you discover Nimbledroid?

Another concern is reproducing issues and/or investigating potential "infra" problems. We go to great pains to make sure our tests can be repeated, that there are records of infra changes that might affect performance, etc. If Nimbledroid reports a regression on a certain build and the cause is non-obvious, is there enough visibility into what they are doing for someone at Mozilla to debug?

I like the idea of leveraging external tools: less for us to build and maintain. I'm just offering some cautions here.

Comment 2

[Chenxia Liu [:liuche]]

Attachment: Screenshot: Nimbledroid overview

These are some of the results from uploading our opt build from 3/25 to Nimbledroid.

Some metrics (see attachment for others, incl breakdown of file size, startup time, etc):
1.3 sec for partial startup (browser chrome loaded, no content)
31,389 methods
38.1 MB


The processing is pretty fast, they had results within 10 minutes. Whatever heuristics they have for startup doesn't work with Fennec though, probably because they can't account for Gecko startup.

Comment 3

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

At droidcon, they mentioned their metric for startup is when the UI thread becomes idle.

We may be able to contact them, let them know our situation, and they might be able to adjust the tools for us.

Comment 4

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Geoff Brown [:gbrown] from comment #1)
> I imagine it would be easy to modify our build jobs to upload builds to
> Nimbledroid. Would we try to get results back from Nimbledroid and into
> Perfherder, or would we just track changes and view results on Nimbledroid?

I spoke to them at Droidcon and they mentioned they have a REST API to download the data (but it wasn't released yet).

> If using Nimbledroid, I wonder if that would decrease the "visibility" of
> results: If you are a platform developer who doesn't normally work with the
> mobile team but is wondering how a recent change affected performance on
> Android, how would you discover Nimbledroid?

We could supply links to Nimbledroid on treeherder/perfherder.

> Another concern is reproducing issues and/or investigating potential "infra"
> problems. We go to great pains to make sure our tests can be repeated, that
> there are records of infra changes that might affect performance, etc. If
> Nimbledroid reports a regression on a certain build and the cause is
> non-obvious, is there enough visibility into what they are doing for someone
> at Mozilla to debug?

I think this is a fair concern. However, perhaps there is a benefit to the fact that someone at Mozilla doesn't need to be the one debugging this issue. ;) It's uncomfortable not knowing all the parts but it could save us resources.

Comment 5

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

NI margaret to check out the file Chenxia uploaded to see if it's useful.

Comment 6

[:Margaret Leibovic]

(In reply to Michael Comella (:mcomella) from comment #5)
> NI margaret to check out the file Chenxia uploaded to see if it's useful.

This seems like it could be useful, but at gbrown pointed out, it does overlap with tools we already have.

My question would be, what problem are we trying to solve with this data? I could see this being useful if we can have a reliable way to measure startup time regressions, but if we're not sure about the accuracy of this data, I'd be hesitant.

I defer to gbrown, since his team would be the one doing the work to actually integrate this into our systems if we choose to go down this road.

Comment 7

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.