595021 - Designs for add-ons performance advisory dashboard

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Nick Nguyen [:osunick]]

Spec:
https://docs.google.com/document/edit?id=1CvpoVd7tLvjM9OtGgwq-KFX-iT2-A8L72rSF7TLrksA&hl=en&authkey=CJaXjugI

This is the dashboard where users and developers can see the add-ons that have > 20% startup time impact on their current platform.  See spec for janky mock.

Comment 1

[Chris Howse [:chowse]]

First drafts:

http://people.mozilla.com/~chowse/drop/amo/performance/v1/03_leaderboard.png
http://people.mozilla.com/~chowse/drop/amo/performance/v1/03_leaderboard_alt.png

* Copy is obvious missing, and will be added shortly.

* The 'For Developers' block should link to any documentation we have on add-on
  performance and, eventually, a way to run the performance tests on your
  own add-ons.

* The hourglass icon used in the alt mockup is from this iconset:
  http://www.icons-land.com/vista-base-software-icons.php
  Not free, but royalty-free for commercial use.

Comment 2

[Nick Nguyen [:osunick]]

This is great, and I have a slight preference for the hourglass version.  I think it does a good job of showing performance, I only wonder if we should use a more hostile color, like red, for the bar.  Also, we should have some text that says "longer bars are slower" or something to that effect.

Comment 3

[Dave Garrett]

(In reply to comment #2)
> I only wonder if we should use a more hostile color, like red, for the bar.

Extension of that idea: base bar color on size. Set the color to be red at 100%, yellow at 20% (or other minimum), and blend in between. Shades of orange in the middle as a result with increasing "hostility" with percentage.

Comment 4

[Chris Howse [:chowse]]

Final mock-up:

http://people.mozilla.com/~chowse/drop/amo/performance/v2/03_leaderboard.png

* Added more copy. Feel free to review and add comments.

* Added a 'longer bars = slower' hint to the table.

* Replaced the blue bars w/ red bars. I liked Dave's idea of a yellow-to-red
  gradation, but apparently the Top 20 are all well over 100%, so I don't think
  it'll improve readability much in this case.

* Added a collapsible section for add-ons that follow the top 10-20. In the
  mocks, I put the split between the Top 10 and Top 50, but we could adjust it
  to any amount (say, Top 20 / Top 100) once we decide how much to include.
  We could also make the entire list available as a CSV, but I don't think that
  belongs in the first iteration.

* Dropped in a replacement icon I found on Veer (could you guess?). I'm going
  to defer on the hourglass icon until a.) we get a decision on whether we can
  include commercial icons in our source or b.) we get someone internal to
  create a replacement.

Comment 5

[Nick Nguyen [:osunick]]

<3

Comment 6

[Dave Garrett]

(In reply to comment #4)
> * Replaced the blue bars w/ red bars. I liked Dave's idea of a yellow-to-red
>   gradation, but apparently the Top 20 are all well over 100%, so I don't think
>   it'll improve readability much in this case.

Then you could set max red for whatever the top happens to be and min yellow or orange for whatever the lowest on the extended list of top 50 or 100 happens to be, then vary through in between for the others in the list. But if the top list is really this bad, then as you say, it might not help.

One note I think is important: it always says "n% slower". I think it needs to explicitly say what is being measured to be slower, e.g. "n% slower startup".

Comment 7

[Chris Howse [:chowse]]

(In reply to comment #6)
> Then you could set max red for whatever the top happens to be and min yellow or
> orange for whatever the lowest on the extended list of top 50 or 100 happens to
> be, then vary through in between for the others in the list.

I'm not sure what new information this would provide, as the list will already be sorted by severity of slowdown and the bars will indicate the relative degree. Colors are good for highlighting outliers or indicating when a threshold is crossed, but unless we set our ranges incredibly wide or highly skewed, I don't see this providing a useful indicator. Of course, we can revisit this once we see the data actually plotted out.


> But if the top list is really this bad, then as you say, it might not help.

You can have a look yourself:
http://brasstacks.mozilla.com/couchdb/addonperf/_design/perfresults/overview.html


> One note I think is important: it always says "n% slower". I think it needs to
> explicitly say what is being measured to be slower, e.g. "n% slower startup".

I hoped the title and intro text would be enough to clarify what the "% slower" indicated. Fitting 'slower startup' in every row might be difficult, but I'll see if I can make it clearer.

Comment 8

[Dave Garrett]

(In reply to comment #7)
> Of course, we can revisit this once we see the data actually plotted out.

Agreed.

> You can have a look yourself:
> http://brasstacks.mozilla.com/couchdb/addonperf/_design/perfresults/overview.html

Oh, thanks for that. One problem, though: no baseline clean profile measurement is given. (unless I'm missing where it is) Checking the CSV for Windows I see it gets all the way down to 400, then a pile of undefineds with a sprinkling of nulls.

I see that I get a score of 430 for Flagfox, though I don't know which version this was tested using. Only +30ms doesn't sound too bad for what I need to do on startup, and I think that number will be lower (depending on how you measure) under Firefox 4 due to async loading for fetching my DB. This, leads me to a new issue with the current mockup that I just realized: it needs to list all extension versions that were tested and also what Firefox version was used for testing. (mockup doesn't have a baseline problem, as it's using percentages of baseline above baseline)