1181822 - Add an A/B test to for low-res tiles vs. fading tiles

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Toolbar, defect)

Description

[James Willcox (:snorp) ([e-mail]) (he/him)]

We could try only doing this in Nightly to start with. The reasoning here is:

1) We have a lot of issues with low-res tiles appearing when they shouldn't. The gfx folks are looking into this stuff now, but...

2) It seems like it may be more trouble than it's worth.

3) With bug 1178376 applied, which allows fading in new tiles, checkerboarding feels a little less jarring.

I propose we try the fading tiles stuff on Nightly to see how folks like it.

Comment 1

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: Disable low-precision tiles on Android, enable fading-in of tiles

Comment 2

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Gotta screenshot? :D

Comment 3

[Kevin Brosnan [Ex-Mozilla]]

I suspect you'll want a video or a build. Screenshot might not be that interesting.

Comment 4

[Brad Lassey [:blassey] (use needinfo?)]

I'm testing on CNN and the low-res tiles are much better

Comment 5

[James Willcox (:snorp) ([e-mail]) (he/him)]

Here's a build with this enabled: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-cb0e9a97ce2b/try-android-api-11/fennec-42.0a1.en-US.android-arm.apk

Comment 6

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Tested this on my N6 and if I'm understanding this correctly, I like the fading in more. 

Even if through measurement, loading the lower res tiles is slightly faster, I think there's something to be said about perceived performance here. It even feels smoother to me.

Comment 7

[Kevin Brosnan [Ex-Mozilla]]

On single core devices (Nexus S Android 4.0) this just does not work. The tiles don't fade in just pop.

Comment 8

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: A screen recording showing the current behavior on nytimes.com, Nexus 9

Comment 9

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: Video showing the fading tiles on nytimes.com, Nexus 9

Comment 10

[James Willcox (:snorp) ([e-mail]) (he/him)]

Ignore the weird urlbar issue on the fading one, that seems to be some regression in Nightly when using synthetic input events.

Comment 11

[James Willcox (:snorp) ([e-mail]) (he/him)]

We want to use an A/B test to determine if replacing the low-res tiles with fading ones will improve the experience.

Comment 12

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: Add A/B experiment for low-res tiles vs. fading on Android

Comment 13

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 8659942 [details] [diff] [review]
Add A/B experiment for low-res tiles vs. fading on Android

This looks like a good approach. I would use ExperimentHelper for Gecko-based pref experiments. Most Java/UI experiments will have the code in to UI code itself. Not to say we might not find a bigger use for ExperimentHelper.

I would rename runABExperiments() -> setup()

This is a good start.

If you add some NIGHTLY_BUILD support in there, I can add the segmentation to the switchboard server and we can make this live.

Comment 14

[Mark Finkle (:mfinkle) (use needinfo?)]

Another thing we should think about is the metrics for the experiment. What can we measure to see if we get a positive improvement. We are gearing up to get churn metrics, but those can be difficult to correlate back to a specific experiment. We'd like some simple data we can send to telemetry.

Comment 15

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

In the future, we could even do further this experiment by looking at more than just fading vs low res.

From convos with snorp on IRC, for example, we could test different ways to optimize perceived perforance through changes in fading duration, linear vs non-linear fading tweens, etc.

Comment 16

[James Willcox (:snorp) ([e-mail]) (he/him)]

I've revived the code that makes fading tiles work, and I would like to turn it on by default in Nightly only. I am not sure what benefit we would really get from a switchboard experiment, and I would like to get the most testing possible with the fading tiles. Anthony does this sound ok to you?

Comment 17

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #16)
> I've revived the code that makes fading tiles work, and I would like to turn
> it on by default in Nightly only. I am not sure what benefit we would really
> get from a switchboard experiment, and I would like to get the most testing
> possible with the fading tiles. Anthony does this sound ok to you?

Sounds fine by me. Let's make sure we're letting everyone else know too (like when we turned on gecko carets for Nightly users) so we can get more feedback :D

Comment 18

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/82d27079b8f1

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/82d27079b8f1

Comment 20

[Barbara Bermes [:barbara] - NI please!]

To be honest, as per Anthony's comment in 6, I'm having issues being persued that this is better performance. 

It's been more "annoying" and disturbing to me to get the black screen fading in and out and stopping me from "thinking" I'm continuing my browsing experience. Obviously being subjective here, and I'd like to get more users giving feedback.