1434093 - Awesomebar popup A/B testing

Status: RESOLVED INCOMPLETE

(Firefox :: Address Bar, enhancement, P2)

Description

[Drew Willcoxon :adw]

Attachment: Flexible Search Dropdown.pdf

UX would like the ability to do some A/B testing with the awesomebar popup.  Attached is a PDF from Eric that shows the things that should be tweakable.  I guess this bug should be a meta bug, with other bugs filed for each piece that we decide to implement.  A big portion of these tweaks are just CSS changes.

Comment 1

[Panos Astithas (he/him) [:past] (please ni?)]

Bumping to P1 as it is a priority for Q1 and the work should start in the next few days or weeks.

Comment 2

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

The hardest point imo is [1.4-2], I doubt our current AC architecture allows something like that, without *huge* hacks. Maybe one day, when we have a grid-flexbox html panel.
Most the rest should be feasible by setting on each result an attribute based on the result type, that for now could just be the same strings as the buckets names (so we may have a type=general, type=search...) and potentially using selectors for first-of-type.

Comment 3

[Panos Astithas (he/him) [:past] (please ni?)]

Yes, this was already brought up. We should proceed with the understanding that anything as hard as that will be followup work.

Comment 4

[Eric Pang [:epang] UX]

Attachment: Flexible Search Dropdown.pdf

Updated the pdf to be in order of priority.  Also added strings for subheadings.

Comment 5

[Drew Willcoxon :adw]

Eric, a couple of comments/questions.

(1) We already have a system in place related to 1.3.1, reordering result types.  It may not be as flexible as you need, though.  I'll describe it.

Currently we divide the awesomebar results into buckets, or sections.  There are four possible buckets: heuristic, general, suggestion, and extension.

Heuristic is always the first bucket, and it only has one result, the first result.  It shows what will happen when you press the return key.  e.g., when you type "foo", it'll show "foo -- Search with Google".  When you type nytimes.com, it'll show "http://nytimes.com/ -- Visit".

General is your bookmarks and history.  We don't distinguish between the two inside this bucket.

Suggestion is search suggestions from your current search engine.

Extension is any results from your installed extensions.

We can change the ordering of buckets and their size on the fly by setting the browser.urlbar.matchBuckets hidden preference, except that the heuristic bucket (with its one result) is hard-coded to always come first, and the extension bucket is hard-coded to always come second.  You probably don't have any extensions that add results to the awesomebar though, so you probably never see the extensions bucket.  If you want to change the hard-coded placements of those two buckets, we'll have to make a code change.

How does that system meet your needs?

(2) Do you have thoughts on how this A/B testing should be delivered to users?  Shield studies or extensions or something like that?  Or do you want to experiment in the product itself from release to release?  From a technical/implementation POV, I'm imagining that these CSS tweaks could be controlled via hidden preferences, and then the question is, how are those preferences changed?

Comment 6

[Drew Willcoxon :adw]

Just a status update: Eric and Philipp are going to discuss this and then get back to us.

Comment 7

[Eric Pang [:epang] UX]

(In reply to Drew Willcoxon :adw from comment #5)
> Eric, a couple of comments/questions.
> 
> (1) We already have a system in place related to 1.3.1, reordering result
> types.  It may not be as flexible as you need, though.  I'll describe it.
> 
> Currently we divide the awesomebar results into buckets, or sections.  There
> are four possible buckets: heuristic, general, suggestion, and extension.
> 
> Heuristic is always the first bucket, and it only has one result, the first
> result.  It shows what will happen when you press the return key.  e.g.,
> when you type "foo", it'll show "foo -- Search with Google".  When you type
> nytimes.com, it'll show "http://nytimes.com/ -- Visit".
> 
> General is your bookmarks and history.  We don't distinguish between the two
> inside this bucket.
> 
> Suggestion is search suggestions from your current search engine.
> 
> Extension is any results from your installed extensions.
> 
> We can change the ordering of buckets and their size on the fly by setting
> the browser.urlbar.matchBuckets hidden preference, except that the heuristic
> bucket (with its one result) is hard-coded to always come first, and the
> extension bucket is hard-coded to always come second.  You probably don't
> have any extensions that add results to the awesomebar though, so you
> probably never see the extensions bucket.  If you want to change the
> hard-coded placements of those two buckets, we'll have to make a code change.
> 
> How does that system meet your needs?

Can we change the appearance of a heuristic result? For example, adding a subheading that appears on top of it.
If we're able to change the visual presentation of the heuristic result I think this system might be fine :).

Also, can you give me an example of an extension that shows up in the extension bucket? I want to see how it’s displayed. How often does this occur?

> 
> (2) Do you have thoughts on how this A/B testing should be delivered to
> users?  Shield studies or extensions or something like that?  Or do you want
> to experiment in the product itself from release to release?

This will more likely be tested in shield studies/extensions.  This keeps the implementation from needing to be perfect and polished. 

> From a technical/implementation POV, I'm imagining that these CSS tweaks could be
> controlled via hidden preferences, and then the question is, how are those
> preferences changed?

I imagine hidden prefs as well, but with more data points to controls part of the ui such as divider line dimensions, text colours, etc.  It might be far too much to ask, but it would be amazing to have a live visualization as we change the values.

Comment 8

[Drew Willcoxon :adw]

> Can we change the appearance of a heuristic result? For example, adding a
> subheading that appears on top of it.
> If we're able to change the visual presentation of the heuristic result I
> think this system might be fine :).

Changes that can be done solely with CSS are simple, things like color, background, font, size, spacing, borders/lines.  Adding subheadings is more of a structural change, and those are a lot harder.  (Or maybe they could be done with :before or :after?  We would need to try to see.)  That also includes grid-like layouts.  As Panos said in comment 3, if possible we'd like to leave structural changes as a follow-up and do the easier CSS tweaks first.

One exception is the buckets as I mentioned, since we have an existing system for that.  We can't add subheadings with that.

> Also, can you give me an example of an extension that shows up in the
> extension bucket? I want to see how it’s displayed. How often does this
> occur?

I found one here: https://github.com/mdn/webextensions-examples/blob/master/firefox-code-search/README.md

I'll attach a couple of screenshots and an xpi you can test yourself.

I don't know how many people have extensions that display results.  I would guess very few.

> This will more likely be tested in shield studies/extensions.  This keeps
> the implementation from needing to be perfect and polished. 

Well, I'm not sure about that.  We would be shipping the hooks for this in Firefox itself, so those hooks and their "infrastructure" need to be polished.  For these CSS tweaks at least, there would be little difference technically between the appearance that ships and tweaks turned on via an A/B test.  Maybe it's possible for an extension to make radical structural changes?  In that case, you'd be right.  But it would have to be an old-style non-web-extension extension, and I don't think those are allowed anymore?  And probably not with Shield studies anyway?

> I imagine hidden prefs as well, but with more data points to controls part
> of the ui such as divider line dimensions, text colours, etc.  It might be
> far too much to ask, but it would be amazing to have a live visualization as
> we change the values.

Is the purpose of this to do A/B testing with users in the field, or is it just for UX people to explore ideas themselves?

Comment 9

[Drew Willcoxon :adw]

Attachment: Extension results after typing "cs foo"

Comment 10

[Drew Willcoxon :adw]

Attachment: Extension results after typing "cs foo" and arrowing down

Comment 11

[Drew Willcoxon :adw]

Attachment: firefox-code-search omnibox extension example

To install this, open about:debugging and click Load Temporary Add-on.  To use it, type "cs foo" in the awesomebar.

More info here:

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Temporary_Installation_in_Firefox
https://github.com/mdn/webextensions-examples/blob/master/firefox-code-search/README.md

Comment 12

[Eric Pang [:epang] UX]

(In reply to Drew Willcoxon :adw from comment #8)
> > Can we change the appearance of a heuristic result? For example, adding a
> > subheading that appears on top of it.
> > If we're able to change the visual presentation of the heuristic result I
> > think this system might be fine :).
> 
> Changes that can be done solely with CSS are simple, things like color,
> background, font, size, spacing, borders/lines.  Adding subheadings is more
> of a structural change, and those are a lot harder.  (Or maybe they could be
> done with :before or :after?  We would need to try to see.)  That also
> includes grid-like layouts.  As Panos said in comment 3, if possible we'd
> like to leave structural changes as a follow-up and do the easier CSS tweaks
> first.

Yep, this works for me!
 

> I found one here:
> https://github.com/mdn/webextensions-examples/blob/master/firefox-code-
> search/README.md
> 
> I'll attach a couple of screenshots and an xpi you can test yourself.
> 
> I don't know how many people have extensions that display results.  I would
> guess very few.

Thanks for all the info around this!  If not many users have extensions that display results this isn't a problem. A little bit odd that the extension icon is where the favicon is as opposed to where the bookmark, open tab icon is located.  Ideally the extension icon (puzzle piece) would be the categorizer (like a bookmark) and the branded extension icon would be in the space of the favicon.  Anyways this isn't part of the bug here :). 

> > This will more likely be tested in shield studies/extensions.  This keeps
> > the implementation from needing to be perfect and polished. 
> 
> Well, I'm not sure about that.  We would be shipping the hooks for this in
> Firefox itself, so those hooks and their "infrastructure" need to be
> polished.  For these CSS tweaks at least, there would be little difference
> technically between the appearance that ships and tweaks turned on via an
> A/B test.  Maybe it's possible for an extension to make radical structural
> changes?  In that case, you'd be right.  But it would have to be an
> old-style non-web-extension extension, and I don't think those are allowed
> anymore?  And probably not with Shield studies anyway?

Makes sense about being polished if the hooks will be shipping in Firefox. I'm not sure if we've figured out how we want to test yet.  So hopefully that info comes soon.

> Is the purpose of this to do A/B testing with users in the field, or is it
> just for UX people to explore ideas themselves?

I think it's a little of both but mainly A/B testing with users.  We'll most likely come up with compositions we want to test and recreate them with the prefs.

Comment 13

[Shilpi Gupta [:shilpi]]

Drew, can you please provide the latest status on this bug? 

Thanks
Shilpi

Comment 14

[Drew Willcoxon :adw]

I'm not sure.  Eric said this in his last comment:

(In reply to Eric Pang [:epang] UX from comment #12)
> I'm not sure if we've figured out how we want to test yet.  So hopefully
> that info comes soon.
> 
> ...
> 
> I think it's a little of both but mainly A/B testing with users.  We'll most
> likely come up with compositions we want to test and recreate them with the
> prefs.

So I don't think it's 100% clear what UX wants yet.  I guess I need to start looking into setting up a shield study at least.

Comment 15

[Shilpi Gupta [:shilpi]]

Update: 

Eric, Drew, Panos and I met this afternoon. We agreed to implement the layout capabilities using prefs so that they can be available to test in a wide variety of ways (shield, user testing, browser builds etc). I will check with relman if we can be part of 60 else we can uplift to 60.

Comment 16

[Shilpi Gupta [:shilpi]]

This bug is delayed and is a P2 per the updated backlog. Aiming for 61/62

Comment 17

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Plans changed, we'll run various experiments in the future, but this won' track that work.