All users were logged out of Bugzilla on October 13th, 2018

add support for perf testing packaged apps to reviewer dashboard

RESOLVED INCOMPLETE

Status

P3
normal
RESOLVED INCOMPLETE
5 years ago
4 years ago

People

(Reporter: rhelmer, Assigned: rhelmer)

Tracking

Avenir
Points:
---

Details

(Assignee)

Description

5 years ago
I have a patch for zamboni/marketplace to submit startup perf testing requests to the "eddy" web service (which uses b2gperf, used for testing gaia apps) and also to pull results from datazilla.mozilla.org

This has been something we've been working in the webtools (now web engineering) team along with the a*team folks for a few quarters now. I have a basic MVP ready, has some rough edges but I'd like to get it reviewed and landed by the end of the month (if at all possible) - it's fairly small and self-contained.
(Assignee)

Updated

5 years ago
Blocks: 918398
I'm not clear where we'd show the perf testing results (link?) in the reviewer tools - it would depend if this is a bulk thing or something that is checked only per app. Also, under what circumstances it would be used.
(Assignee)

Comment 2

5 years ago
(In reply to Andrew Williamson [:eviljeff] from comment #1)
> I'm not clear where we'd show the perf testing results (link?) in the
> reviewer tools - it would depend if this is a bulk thing or something that
> is checked only per app. Also, under what circumstances it would be used.

Here's a PR: https://github.com/mozilla/zamboni/pull/1160

I put a graph in the app review page, along with a button (which does perf tests on demand).
(Assignee)

Comment 3

5 years ago
(In reply to Andrew Williamson [:eviljeff] from comment #1)
> Also, under what circumstances it would be used.

The idea is to provide a way to get objective startup time results on target hardware - b2gperf reboots the phone and starts the app up 30 times, so if the app doesn't start up or is very slow on average on lowish-end production devices, we can let developers know.

There has been interest in exposing this to app developers themselves, which is doable (we're allocating more phones right now and would need to get the queuing right, I don't think it'll be a problem though).

I am going to talk about it at the apps show-and-tell today too.
(Assignee)

Comment 4

5 years ago
OK some feedback I got at the apps show-and-tell was to go ahead and expose this to developers, since it's probably more actionable for them than anyone else.

Long-term I'd really like to just have perf tests run automatically rather than on-demand like this, but there's concern from the team that runs the test devices so it might be good to just stick with the button for now (we could possibly put it under developer's control though!)
(Assignee)

Comment 5

5 years ago
(In reply to Robert Helmer [:rhelmer] from comment #4)
> OK some feedback I got at the apps show-and-tell was to go ahead and expose
> this to developers, since it's probably more actionable for them than anyone
> else.

I am thinking about exposing this on the /app/{appname}/statistics/ page.
(In reply to Robert Helmer [:rhelmer] from comment #4)
> OK some feedback I got at the apps show-and-tell was to go ahead and expose
> this to developers, since it's probably more actionable for them than anyone
> else.

Yes, I agree.  It would also be good to have something like the standalone validator so developers can submit arbitrary versions under development to check performance (and hopefully to see how good their improvements are)
Btw, not sure about how this is (or is going to be) implemented, but it should be an API
firefox-marketplace-api.readthedocs.org 

The validation pattern [http://firefox-marketplace-api.readthedocs.org/en/latest/topics/submission.html] is probably most similar if its going to be something that might take a few minutes to complete.
Priority: -- → P3
(Assignee)

Comment 8

5 years ago
(In reply to Andrew Williamson [:eviljeff] from comment #7)
> Btw, not sure about how this is (or is going to be) implemented, but it
> should be an API
> firefox-marketplace-api.readthedocs.org 
> 
> The validation pattern
> [http://firefox-marketplace-api.readthedocs.org/en/latest/topics/submission.
> html] is probably most similar if its going to be something that might take
> a few minutes to complete.

Oh, I really like this idea! I will do this.
(Assignee)

Comment 9

5 years ago
It will take a few minutes to complete - you fire off a request and get a token back, and can use that token to check on status later. Ideally we'd just fire off a request when apps are uploaded and not wait for it, just populate a graph on the statistics page (or somewhere) as results are ready, it doesn't need to be blocking in the way that the validator is IMHO.

Comment 10

4 years ago
Didn't get done, if we ever plan to do this again, let's re-open.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.