The default bug view has changed. See this FAQ.

Add A/B experiment for compact tabs

VERIFIED FIXED in Firefox 53

Status

()

Firefox for Android
General
VERIFIED FIXED
3 months ago
a month ago

People

(Reporter: sebastian, Assigned: sebastian)

Tracking

(Depends on: 1 bug)

unspecified
Firefox 53
All
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox53 verified, firefox54 verified)

Details

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments)

(Assignee)

Description

3 months ago
Create an A/B experiment for the new compact tabs setting and add telemetry that is needed to make a decision.

* 50% of the users should get the existing tabs tray layout. The other 50% should have compact tabs enabled by default.
* Determine which data we need to make a decision about the new default (compact tabs or the existing layout).
(Assignee)

Comment 1

3 months ago
Barbara, Anthony, I'm wondering..

* .. do we want to hide the setting or should users in the experiment still be able to switch between both modes?
* .. what exactly do we want to measure? Of course I can add additional probes to the tabs tray, but I'm not sure how they should help us find a new default.
Flags: needinfo?(bbermes)
Flags: needinfo?(alam)

Comment 2

3 months ago
(In reply to Sebastian Kaspari (:sebastian) from comment #1)
> Barbara, Anthony, I'm wondering..
> 
> * .. do we want to hide the setting or should users in the experiment still
> be able to switch between both modes?

I think they should still be able to switch. This would be a really great metric to track too (given which default they had).

> * .. what exactly do we want to measure? Of course I can add additional
> probes to the tabs tray, but I'm not sure how they should help us find a new
> default.

Good question! Perhaps we can ask for some help from Gemma if she has some quick suggestions here but I think the problem we're trying to solve might just be about personalisation/customization. So it might be a matter of preference based on the way different users use their browser. 

The side benefit of this would be messaging. It's a nice differentiation factor for ourselves as well and something that we can talk loudly about. But I'm not sure that helps with setting one as a default over the other... I wonder if there's a strong case for either here..
Flags: needinfo?(alam)
(Assignee)

Comment 3

3 months ago
Added new experiment with name "compact-tabs" to switchboard config (buckets 0-50).
(Assignee)

Comment 4

3 months ago
(In reply to Anthony Lam (:antlam) from comment #2)
> I think they should still be able to switch. This would be a really great
> metric to track too (given which default they had).

The problem with the preference is that if we allow the users to set it themselves then we can't change the experiment configuration anymore once it's deployed and the users change the setting (We shouldn't override this once set manually).
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 8

3 months ago
mozreview-review
Comment on attachment 8820357 [details]
Bug 1323952 - Add compact tabs experiment.

https://reviewboard.mozilla.org/r/99854/#review100322
Attachment #8820357 - Flags: review?(ahunt) → review+

Comment 9

3 months ago
mozreview-review
Comment on attachment 8820358 [details]
Bug 1323952 - Tabs panel: Update default based on compact tabs experiment.

https://reviewboard.mozilla.org/r/99856/#review100326
Attachment #8820358 - Flags: review?(ahunt) → review+

Comment 10

3 months ago
mozreview-review
Comment on attachment 8820359 [details]
Bug 1323952 - Preferences: Set default 'compact tabs' value based on experiment.

https://reviewboard.mozilla.org/r/99858/#review100328
Attachment #8820359 - Flags: review?(ahunt) → review+
(Assignee)

Updated

3 months ago
Attachment #8820357 - Flags: review?(cnevinchen)
Attachment #8820358 - Flags: review?(cnevinchen)
Attachment #8820359 - Flags: review?(cnevinchen)

Comment 11

3 months ago
mozreview-review
Comment on attachment 8820357 [details]
Bug 1323952 - Add compact tabs experiment.

https://reviewboard.mozilla.org/r/99854/#review100454
Attachment #8820357 - Flags: review?(cnevinchen) → review+

Comment 12

3 months ago
mozreview-review
Comment on attachment 8820359 [details]
Bug 1323952 - Preferences: Set default 'compact tabs' value based on experiment.

https://reviewboard.mozilla.org/r/99858/#review100458
Attachment #8820359 - Flags: review?(cnevinchen) → review+

Comment 13

3 months ago
mozreview-review
Comment on attachment 8820358 [details]
Bug 1323952 - Tabs panel: Update default based on compact tabs experiment.

https://reviewboard.mozilla.org/r/99856/#review100460
Attachment #8820358 - Flags: review?(cnevinchen) → review+

Comment 14

3 months ago
Pushed by s.kaspari@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/c1224e1c7198
Add compact tabs experiment. r=ahunt,nechen
https://hg.mozilla.org/integration/autoland/rev/fbb8ff8e9ad9
Tabs panel: Update default based on compact tabs experiment. r=ahunt,nechen
https://hg.mozilla.org/integration/autoland/rev/03bd43cec49f
Preferences: Set default 'compact tabs' value based on experiment. r=ahunt,nechen

Comment 15

3 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/c1224e1c7198
https://hg.mozilla.org/mozilla-central/rev/fbb8ff8e9ad9
https://hg.mozilla.org/mozilla-central/rev/03bd43cec49f
Status: NEW → RESOLVED
Last Resolved: 3 months ago
status-firefox53: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 53
(In reply to Sebastian Kaspari (:sebastian) from comment #1)
> Barbara, Anthony, I'm wondering..
> 
> * .. do we want to hide the setting or should users in the experiment still
> be able to switch between both modes?

Still visible. Not quite sure if I understand comment #4's issue?


> * .. what exactly do we want to measure? Of course I can add additional
> probes to the tabs tray, but I'm not sure how they should help us find a new
> default.

Anthony, since you mostly pushed for this UX change, what was the main driver for it? What's your hypothesis for doing this? i.e. the Why? Is the question around should we make this eventually the default setting, or is the compact tabs view even more engaging at all?

To me, basic decision drivers / metrics should be around:

a) Do people with compact tabs enabled load more URLs from tabs tray than the ones who don't have it enabled
b) Do people with compact tabs enabled retain better 
c) How many people in the control group discover / set the setting to "compact tabs" once visible to them

So we need probes 
- for setting toggle
- to keep track of experiment group
- to keep track of action on tab, i.e. load (close etc.)
Flags: needinfo?(s.kaspari)
Flags: needinfo?(bbermes)
Flags: needinfo?(alam)
(Assignee)

Comment 17

3 months ago
(In reply to Barbara Bermes [:barbara] - NI please! from comment #16)
> (In reply to Sebastian Kaspari (:sebastian) from comment #1)
> > Barbara, Anthony, I'm wondering..
> > 
> > * .. do we want to hide the setting or should users in the experiment still
> > be able to switch between both modes?
> 
> Still visible. Not quite sure if I understand comment #4's issue?

The current experiment setup is to change the default for 50% of the (for now: Nightly) users. That's what we want to test / maybe change, right? That's why I was asking if we want to hide/show the setting additionally.

> a) Do people with compact tabs enabled load more URLs from tabs tray than
> the ones who don't have it enabled

Do we really expect a different tabs tray layout to change the URL loading behavior? 

> b) Do people with compact tabs enabled retain better 

From all the factors that play into that .. it feels like the contribution of this feature seems insignificant.

> c) How many people in the control group discover / set the setting to
> "compact tabs" once visible to them

It seems like we are thinking of different experiments: My setup is changing the default - You just want to expose the setting - Correct?
Flags: needinfo?(s.kaspari)
(Assignee)

Updated

3 months ago
Depends on: 1325613
(In reply to Barbara Bermes [:barbara] - NI please! from comment #16)
> Anthony, since you mostly pushed for this UX change, what was the main
> driver for it? What's your hypothesis for doing this? i.e. the Why? Is the
> question around should we make this eventually the default setting, or is
> the compact tabs view even more engaging at all?

This was a part of the original Tabs tray design refresh back in bug 1158277. It was left off the list (unfinished) because we had underestimated the technical debt in the code for this part of the product. So we're really just finishing off the original redesign.

IIRC, our original intention for redesigning the tabs tray was multi-fold:

1) Fix the issues with the original design of the tabs tray - 
  a) the tabs preview proportions were dated (device/screen sizes have been constantly changing) 
  b) they did not accurately reflect the proportions of what the user was seeing
  c) most users had more tabs than could fit on the screen at one time

2) Update a high-visibility portion of our products UI to stay competitive (and just not look dated) with other products on the Play Store

3) Match iOS and bring increased cross-platform consistency within the Firefox ecosystem (and iOS has this on by default).

4) Match Tablet experiences too

> To me, basic decision drivers / metrics should be around:
> 
> a) Do people with compact tabs enabled load more URLs from tabs tray than
> the ones who don't have it enabled
> b) Do people with compact tabs enabled retain better 
> c) How many people in the control group discover / set the setting to
> "compact tabs" once visible to them
> 
> So we need probes 
> - for setting toggle
> - to keep track of experiment group
> - to keep track of action on tab, i.e. load (close etc.)

I'd love to measure all of these things too. But I would hesitate to hold off on finishing this last part of the redesign just because users don't load more URLs after this change. :)

Of course, if we find that retention numbers are terrible comparatively, or know that people are not loading URLs because their tabs tray looks different, then we should definitely _not_ do this.  

But I believe there's still something to be said about staying up to date and relevant. Front-and-center, showing our users that we're constantly looking to improve the product and matching their Firefox experience and expectations on other platforms where we exist.
Flags: needinfo?(alam)
Sebastian

> The current experiment setup is to change the default for 50% of the (for now: Nightly) users. That's what we want to test / maybe change, right? That's why I was asking if we want to hide/show the setting additionally.

I think I got it now ;) I'd go with showing the setting additionally.

So to confirm, we agree to the following:
- Variant A (control) 50%: no compact tabs on by default, show setting to disable/enable
- Variant B 50%: compact tabs on by default, show setting to disable/enable


Anthony,

>   c) most users had more tabs than could fit on the screen at one time

Do we have a probe for that?

> 3) Match iOS and bring increased cross-platform consistency within the
> Firefox ecosystem (and iOS has this on by default).

Very valid point! 


> I'd love to measure all of these things too. But I would hesitate to hold
> off on finishing this last part of the redesign just because users don't
> load more URLs after this change. :)

Product decision are thankfully not made so black and white ;)

Comment 20

a month ago
Verified on the latest Nightly 54 (2017-27-02) that the "compact-tabs" experiment is added to the switchboard and that the setting is on by default in 50% of the time, on clean installs.
Devices: 
- LG G4 (Android 5.1)
- Oneplus Two (Android 6.0)
Status: RESOLVED → VERIFIED
status-firefox53: fixed → verified
status-firefox54: --- → verified
You need to log in before you can comment on or make changes to this bug.