918562 - add support for perf testing packaged apps to reviewer dashboard

Status: RESOLVED INCOMPLETE

(Marketplace Graveyard :: Reviewer Tools, defect, P3)

Description

[Robert Helmer [:rhelmer]]

I have a patch for zamboni/marketplace to submit startup perf testing requests to the "eddy" web service (which uses b2gperf, used for testing gaia apps) and also to pull results from datazilla.mozilla.org

This has been something we've been working in the webtools (now web engineering) team along with the a*team folks for a few quarters now. I have a basic MVP ready, has some rough edges but I'd like to get it reviewed and landed by the end of the month (if at all possible) - it's fairly small and self-contained.

Comment 1

[Andrew Williamson [:eviljeff]]

I'm not clear where we'd show the perf testing results (link?) in the reviewer tools - it would depend if this is a bulk thing or something that is checked only per app. Also, under what circumstances it would be used.

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Andrew Williamson [:eviljeff] from comment #1)
> I'm not clear where we'd show the perf testing results (link?) in the
> reviewer tools - it would depend if this is a bulk thing or something that
> is checked only per app. Also, under what circumstances it would be used.

Here's a PR: https://github.com/mozilla/zamboni/pull/1160

I put a graph in the app review page, along with a button (which does perf tests on demand).

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to Andrew Williamson [:eviljeff] from comment #1)
> Also, under what circumstances it would be used.

The idea is to provide a way to get objective startup time results on target hardware - b2gperf reboots the phone and starts the app up 30 times, so if the app doesn't start up or is very slow on average on lowish-end production devices, we can let developers know.

There has been interest in exposing this to app developers themselves, which is doable (we're allocating more phones right now and would need to get the queuing right, I don't think it'll be a problem though).

I am going to talk about it at the apps show-and-tell today too.

Comment 4

[Robert Helmer [:rhelmer]]

OK some feedback I got at the apps show-and-tell was to go ahead and expose this to developers, since it's probably more actionable for them than anyone else.

Long-term I'd really like to just have perf tests run automatically rather than on-demand like this, but there's concern from the team that runs the test devices so it might be good to just stick with the button for now (we could possibly put it under developer's control though!)

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #4)
> OK some feedback I got at the apps show-and-tell was to go ahead and expose
> this to developers, since it's probably more actionable for them than anyone
> else.

I am thinking about exposing this on the /app/{appname}/statistics/ page.

Comment 6

[Andrew Williamson [:eviljeff]]

(In reply to Robert Helmer [:rhelmer] from comment #4)
> OK some feedback I got at the apps show-and-tell was to go ahead and expose
> this to developers, since it's probably more actionable for them than anyone
> else.

Yes, I agree.  It would also be good to have something like the standalone validator so developers can submit arbitrary versions under development to check performance (and hopefully to see how good their improvements are)

Comment 7

[Andrew Williamson [:eviljeff]]

Btw, not sure about how this is (or is going to be) implemented, but it should be an API
firefox-marketplace-api.readthedocs.org 

The validation pattern [http://firefox-marketplace-api.readthedocs.org/en/latest/topics/submission.html] is probably most similar if its going to be something that might take a few minutes to complete.

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Andrew Williamson [:eviljeff] from comment #7)
> Btw, not sure about how this is (or is going to be) implemented, but it
> should be an API
> firefox-marketplace-api.readthedocs.org 
> 
> The validation pattern
> [http://firefox-marketplace-api.readthedocs.org/en/latest/topics/submission.
> html] is probably most similar if its going to be something that might take
> a few minutes to complete.

Oh, I really like this idea! I will do this.

Comment 9

[Robert Helmer [:rhelmer]]

It will take a few minutes to complete - you fire off a request and get a token back, and can use that token to check on status later. Ideally we'd just fire off a request when apps are uploaded and not wait for it, just populate a graph on the statistics page (or somewhere) as results are ready, it doesn't need to be blocking in the way that the validator is IMHO.

Comment 10

[Andy McKay]

Didn't get done, if we ever plan to do this again, let's re-open.