Design: Graphs for FN14 GA benchmarks

RESOLVED FIXED

Status

Marketing
Design
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: Jaclyn Fu, Assigned: Ty Flanagan)

Tracking

Details

Attachments

(9 attachments, 2 obsolete attachments)

(Reporter)

Description

6 years ago
Hi John/Tara,

We have some benchmark numbers for Firefox Android that we'd like to make pretty so we can put them into press kits/blog posts etc. We're in a bit of a rush though, would EOD June 11 be possible? 

The higher the numbers the better - how can we make it optically look like this means faster even though the numbers are bigger? 

Graph title: Page load speed 
Subtext: The higher the number the better  

Firefox for Android (latest version): 220
Firefox for Android (previous version): 80
Default Android Browser: 65
Opera: 112
Chrome for Android: 102
Dolphin Browser: 74

Thank you!
(Reporter)

Comment 1

6 years ago
**horizontal bar graph
(Reporter)

Comment 2

6 years ago
Also clarification on the unit of measurements:

Number of unique frames in a capture. Higher is better (higher # of frames = faster rendering = page load speed)
(In reply to Jaclyn Fu from comment #2)
> Also clarification on the unit of measurements:
> 
> Number of unique frames in a capture. Higher is better (higher # of frames =
> faster rendering = page load speed)

Minor correction: this benchmark doesn't have anything to do with page load speed.

"higher # of frames = faster rendering" is perfectly accurate though.
(Reporter)

Comment 4

6 years ago
Thanks for the correction!

Let's move forward with calling this graph "Page rendering" (William - that's accurate?)
(In reply to Jaclyn Fu from comment #4)
> Thanks for the correction!
> 
> Let's move forward with calling this graph "Page rendering" (William -
> that's accurate?)

"Canvas rendering" would be more precise. Page rendering sounds pretty vague to me -- this benchmark measures performance only for a specific type of page.
(In reply to William Lachance (:wlach) from comment #3)
> (In reply to Jaclyn Fu from comment #2)
> > Also clarification on the unit of measurements:
> > 
> > Number of unique frames in a capture. Higher is better (higher # of frames =
> > faster rendering = page load speed)
> 
> Minor correction: this benchmark doesn't have anything to do with page load
> speed.
> 
> "higher # of frames = faster rendering" is perfectly accurate though.

It also occurred to me last night that I could give you numbers in terms of frames per second on this benchmark. The proportions would be approximately the same, but the metric would probably be more familiar to the technical press. Frames per second is commonly used in graphics card benchmarks, for example. e.g. http://www.tomshardware.com/charts/2011-gaming-graphics-charts/Aliens-vs.-Predator-Gamer,2667.html
(Reporter)

Comment 7

6 years ago
ok cool, let's do frames per second. Is it still higher the number the better and measuring canvas rendering?
(In reply to Jaclyn Fu from comment #7)
> ok cool, let's do frames per second. Is it still higher the number the
> better and measuring canvas rendering?

Yep, I'll post the new numbers shortly.
Frames per second benchmarks:

Firefox for Android (previous version): 13.2001638884
Firefox for Android (latest version): 39.273391046
Opera: 21.6094041976
Chrome for Android: 19.6808266828
Dolphin Browser: 14.1638832422
Default Android Browser: 12.0694154442

Comment 10

6 years ago
I've asked Ty to help with this. EOD today isn't really doable, but we'll see what we can get done tmrw.
(Assignee)

Comment 11

6 years ago
Do we need to include which Android OS version was used to obtain these benchmarks?
(Reporter)

Comment 12

6 years ago
William - can you provide what device and version you used? thanks!
(In reply to Jaclyn Fu from comment #12)
> William - can you provide what device and version you used? thanks!

Samsung Galaxy Nexus - Android 4.0.4.
(Assignee)

Comment 14

6 years ago
Created attachment 632417 [details]
Benchmark Graph - FFxAurora - Printable PDF
(Assignee)

Comment 15

6 years ago
Created attachment 632418 [details]
Benchmark Graph - FFxAurora - Printable TIF - CMYK 300dpi
(Assignee)

Comment 16

6 years ago
Created attachment 632419 [details]
Benchmark Graph - FFxAurora - Web 640x480
(Assignee)

Comment 17

6 years ago
Created attachment 632421 [details]
Benchmark Graph - FFxAurora - Web 1280x932

Comment 18

6 years ago
Love the graphic, thanks Ty.

One thought: we should consider changing the title of the graphic...calling it "Canvas rendering speed" actually makes us look pretty slow if you don't read the fine print (the Firefox bar is much larger than the rest, which make it look like we take longer).

In other words, I like the visuals, but we should make sure it's clear to people what they're saying. Could even give it an overall headline like "Firefox for Android is Fast!" (note - that's a terrible headline, don't really use it) and then add a subhead that explains what the graphic is.

Jaclyn, how would you like to position this?
(Reporter)

Comment 19

6 years ago
+1 love the visual!

Agreed, at first glance the perception might be these results show the opposite of what it really is. 

How about something like...

Do more, faster with Firefox for Android 
Canvas rendering performance

Is that lame.
(Reporter)

Comment 20

6 years ago
William - did you use Opera Mini or Opera Browser?

Comment 21

6 years ago
Matej, can you give us a quick copy consult here? See comment #18 and 19 for details.

Thanks!
(Reporter)

Comment 22

6 years ago
Ty - and let's add a note about what site we used (William, doubling checking that we used nyt.com)

"*Results based on using nyt.com on a Samsung Galaxy Nexus on 4.0.4"

Thanks!
On the whole, I don't feel like "canvas rendering speed" is adding a lot as something big and prominent at the top of the image. Could that go at the bottom with the fine print? "Canvas rendering speed based on a Samsung Galaxy Nexus running Android 4.0.4 on nyt.com"

I would actually suggest putting the "Frames Per Second" and all that copy at the top of the graph because that clearly states that a higher number is better.

That way we wouldn't need a subhead, either, and the headline could be one of the following:

Get up to speed with Firefox for Android

Browse faster with Firefox for Android

Get to the Web fast with Firefox for Android


One other question: Is there a reason we're using green for our performance? Seems like we've been using blue lately for mobile. Or, obviously, orange.
Replying to stuff in bulk:

On the graph: The "better page load time" on the bottom of the graph is inaccurate. This measurement doesn't have anything to do with page load time. I'd either take it out or replace it with "smoother animations" (e.g. higher # of frames = faster rendering = smoother animations).

On the page:

This isn't the New York Times, it's a synthetic benchmark I created myself for measuring canvas performance with Eideticker. An example of what the benchmark looks like is here:

http://wrla.ch/eideticker/dashboard/videos/video-1338408277.15.webm

In general this graph would indicate that the new Firefox for Android would have better performance in highly interactive sites and games.

On version of Opera (Opera mini vs. Opera browser):

I used the Opera browser. I don't think this benchmark would actually work on Opera mini.
Oh, and one random comment on the graph: I don't think we need to get down to 4 digits of precision. ;) The results of this test are fairly noisy, and you might get values 1 or 2 different per run. I'd use at most one digit.

e.g.

Firefox for Android (previous version): 13.2
Firefox for Android (newest version): 39.3

etc.
(Reporter)

Comment 26

6 years ago
Thanks!

hm if it's smoother animations, then I don't we can say this graph shows how fast Firefox for Android is. Unless we just say higher # of frames = faster rendering and just leave it at that

Ok Ty - let's not add the bit about nyt.com in the description then. Can we add "Browser" to "Opera"? And also let's simplify the digits

slater/matej - what should we do about the copy about rendering and how to describe?
(In reply to Jaclyn Fu from comment #26)

> hm if it's smoother animations, then I don't we can say this graph shows how
> fast Firefox for Android is. Unless we just say higher # of frames = faster
> rendering and just leave it at that

The technical press/advertising often talks about how "fast" graphics are based on metrics like these. For example, look at the section on "outstanding performance" at http://www.apple.com/imac/performance.html. I'm obviously not an advertising expert, but I don't think it's too much of a stretch to talk about speed here.
(Reporter)

Comment 28

6 years ago
Ok good to know. We can spin it as "performance" then..

What about..

Canvas rendering performance
Frames per second: higher # of frames = faster graphics rendering
I would still keep "canvas rendering performance" out of the header area (I don't think that's going to mean a lot to people), but would make what the graph is representing more prominent (the frames per second stuff).

Based on the last few comments, it sounds like "Get up to speed with Firefox for Android" would still work for the headline.
(Reporter)

Comment 30

6 years ago
I think since this is mainly going to be a Press graphic, they should know what canvas rendering is 

To sum up:

For the note on the bottom,
*Results based on new Eideticker Canvas Benchmark using Samsung Galaxy Nexus on 4.0.4.

Opera -> Opera Browser

Simply digits to just one digit after the decimal is fine
(Reporter)

Comment 31

6 years ago
We also have a second set of benchmark results we'd like to create into a graph as well:

This one measures performance while panning a website while using cnn.com:

Title - Website Panning Performance 
Subtext: Frames per second over capture (higher # of frames = smoother rendering):

Firefox (old): 10.2
Opera Browser: 9.6
Chrome for Android:11.9
Dolphin Browser:14.2
Default Android Browser: 13.9
Firefox for Android (new): 20.5

Thanks!

Comment 32

6 years ago
What's the timing for this 2nd graph? Ideally we'd be given more heads up for this sort of thing.
(Reporter)

Comment 33

6 years ago
It would be great if we could have both graphs in the reviewer's guide...but if it's not possible then can we get it in time for the blog post? (June 26)

Comment 34

6 years ago
Ok. Ty, let us know what's doable. At the least, we should get the first one handed off today.
(Assignee)

Comment 35

6 years ago
Created attachment 633137 [details]
Benchmark Graph - FFxAurora - Canvas Rendering - Web

Please let me know if this covers all the changes you asked for including copy tone.
(Assignee)

Comment 36

6 years ago
Created attachment 633138 [details]
Benchmark Graph - FFxAurora - Website Panning - Web
(Assignee)

Comment 37

6 years ago
Once all changes have been made/approved, i'll go ahead and produce a printable version of each graph
These look great to me! Everything looks technically correct as well. I have just a few final suggestions on clarifying content (I guess it's up to Jaclyn to approve these, this is just IMO):

1. For the first graph, instead of just saying 4.0.4, I would say "Android 4.0.4" (makes it more clear what you're talking about)
2. For the second graph, I'd put down the version of the operating system as well (so: "Results based on performance measures while panning a website using http://www.cnn.com using Samsung Galaxy Nexus on Android 4.0.4")

Comment 39

6 years ago
Thanks Ty!

Jaclyn, please give the signoff as soon as you can so we can get these over to Rhonda for the reviewer's guide.
Just to chime - I agree with William's feedback - think we should add those small descriptors to make it very clear how we got these #'s
(Reporter)

Comment 41

6 years ago
> 1. For the first graph, instead of just saying 4.0.4, I would say "Android
> 4.0.4" (makes it more clear what you're talking about)

+1 

> 2. For the second graph, I'd put down the version of the operating system as
> well (so: "Results based on performance measures while panning a website
> using http://www.cnn.com using Samsung Galaxy Nexus on Android 4.0.4")

+1

Also for the second graph can we label the old version as "Firefox for Android (previous version)" to keep it consistent with the first graph? 

Thanks Ty!
(Assignee)

Comment 42

6 years ago
Created attachment 633180 [details]
Benchmark Graph - FFxAurora - Canvas Rendering - Web
Attachment #633137 - Attachment is obsolete: true
(Assignee)

Comment 43

6 years ago
Created attachment 633181 [details]
Benchmark Graph - FFxAurora - Website Panning - Web
Attachment #633138 - Attachment is obsolete: true
(Assignee)

Comment 44

6 years ago
Created attachment 633182 [details]
Benchmark Graph - FFxAurora - Canvas Rendering - Print
(Assignee)

Comment 45

6 years ago
Created attachment 633183 [details]
Benchmark Graph - FFxAurora - Website Panning - Print
(Reporter)

Comment 46

6 years ago
Hi Ty, spoke to slater about this, but we're getting in some new benchmark data tonight and will need a tight turn around to make into a pretty graph for tomorrow morning

will post in here as soon as we get it, thank you!
(Assignee)

Comment 47

6 years ago
Sounds good. If youre able to get me anything tonight, I can get it created this for reviewing tomorrow in the AM.
(Reporter)

Comment 48

6 years ago
If we can provide the metrics this evening, when's the earliest we can have a review tomorrow morning EDT time? We're up for waking up early to review

Updated

6 years ago
Assignee: tshahian → tyronflanagan
Status: NEW → ASSIGNED
(Assignee)

Comment 49

6 years ago
any luck on getting those metrics tonight?
Status: ASSIGNED → NEW
(Reporter)

Comment 50

6 years ago
Hey Ty, sorry...looks like they're slowly trickling in. Don't wait up for it tonight, they'll be in your inbox when you're up tomorrow? :)

Comment 51

6 years ago
Thanks Ty!

Jaclyn, just to be clear, this is basically just updating the earlier graphic with new data?
(Reporter)

Comment 52

6 years ago
With different data (Javascript benchmark) - but same in the same format as the previous data

Updated

6 years ago
Blocks: 719894
(Reporter)

Comment 53

6 years ago
Created attachment 635505 [details]
Previous JS benchmark example

Hey Ty,

We finally have numbers! Sorry about the delay. This is a different benchmark from the one we previously did, and I have an example of something we did in the past for JS benchmarks. Can we do something similar? Thank you!

We have results from 3 benchmarks: Kraken, Sunspider, V8
Sorting results by Android 2.2 (Froyo), 2.3 (Gingerbread), 4.0 (ICS) 
Firefox for Android vs Stock Browser 
Besides on the axis, we don't need to show the actual numbers of the results

KRAKEN (lower the better)
2.2 
-Firefox: 
-Stock:
2.3
-Firefox: 55721
-Stock: Browser not responsive
4.0 
-Firefox: 34058
-Stock: 41950

SUNSPIDER (lower the better)
2.2
-Firefox:
-Stock: 6836
2.3: 
-Firefox: 3139
-Stock: 5313
4.0
-Firefox: 1687
-Stock: 1875

V8 (higher the better)
2.2
-Firefox: 420
-Stock: 178
2.3
-Firefox: 459
-Stock: 215
4.0
-Firefox: 877
-Stock: 1173

We have a couple of holes we sitll need to fill in but is this enough to get started? Will ping with the remaining data

Thank you!!!! What do you estimate the turnaround to be?
(Assignee)

Comment 54

6 years ago
Just to verify, I'm creating 3 separate graphs correct? (1) KRAKEN; (2) SUNSPIDER; (3) V8

Turnaround is all dependent on how soon you can get me the rest of the data.

Comment 55

6 years ago
I hate to throw a wrench in this now, but seeing that we're only 25% faster, I think that's going to be hard to make look awesome in a graph. Since we are trying to change the conversation and point to Eideticker, I think we should try to create a graph around those results that shows up much faster. Jaclyn - can you update w/Eideticker results here?

Comment 56

6 years ago
I'm obviously delirious - Ty already made those graphs in this bug! Jaclyn/Dave - I assume we want to use the panning one but please confirm and let Ty know if there are any changes to be made. Otherwise, Ty I think we'll just need your help dropping this is the rev guide file.
(Reporter)

Comment 57

6 years ago
Ping Ty - can we drop this into the rev guide?

Comment 58

6 years ago
Adding Val and Tetsuya to access the high res print and web performance graph attachments- there is one for panning and one for canvas

Comment 59

6 years ago
Can we close this?  If so, please close.

Updated

6 years ago
Status: NEW → ASSIGNED
(Reporter)

Updated

6 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.