1030864 - [Search experiment] Notify user about the end of the experiment

Status: VERIFIED FIXED

(Firefox :: Search, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

From bug 1029275, we should notify the user about the end of the experiment.

Q: Is that only for the users from cells 3 and 5 who were notified at the beginning of the experiment? Or does that apply to cells 2 and 4 too?


On a technical side, we will have to tell the experiment manifest to run the experiment a bit longer than what's desired, and let the add-on code decide by itself when to finish it, display the notification (and decide the best moment to do so), and uninstall itself. Also avoid doing that in case of a uninstall/reinstall due to an upgrade. It's a bit more complex for these reasons, but should be doable.

Comment 1

[Javaun Moradi [:javaun]]

Do we need a UX design for this end-of-experiment user notification? Or do we already have it?

Comment 2

[(Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo]

There's a mockup in bug 1029189 about this (https://bugzilla.mozilla.org/attachment.cgi?id=8446642).
This is really just the same thing as at the beginning of the experiment but with a Google logo. I'm very open for other suggestions here, but that should be a pretty safe route to go.

Comment 3

[:Felipe Gomes (needinfo for replies!)]

Same text, even? Just displaying the then-restored Google provider?

Comment 4

[:Felipe Gomes (needinfo for replies!)]

For the exact behavior, here's what I propose:

3 days before the end date of the experiment, the experiment:
 - restores the default search provider
 - resets the counter of number of times the notice has been displayed, allowing it to be shown 5 more times

This means that there will be a 3-day window buffer where users have a chance to be notified about the reset. If a user doesn't open Firefox during that window, the add-on will be uninstalled and the provider reset, but it won't have a chance to notify about the change.

I need to double check this, it might still be possible to do something on uninstallation. But if it isn't, does this sound reasonable?

Comment 5

[(Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo]

Sounds good to me!

I went with the same text because it is my understanding that we don't want to tell people specifically that they've been part of an experiment. I might be wrong about that though. Chad?

Comment 6

[Blake Winton (:bwinton) (:☕️)]

Having just read about the recent Facebook experiment at http://www.theatlantic.com/technology/archive/2014/06/everything-we-know-about-facebooks-secret-mood-manipulation-experiment/373648/, it seems like disclosing to people that they've been part of an experiment might be the ethical thing to do, particularly if we don't notify them at the start.  (Notifying the control group is probably more optional. ;)

Comment 7

[:Felipe Gomes (needinfo for replies!)]

Yeah I had to go with the 3-day buffer
https://hg.mozilla.org/users/felipc_gmail.com/search-experiment/rev/0e1f70de761f

Comment 8

[Marco Mucci [:MarcoM]]

Added to Iteration 33.2
Felipe, can you provide a point value and if the bug should be marked as [qa+] or [qa-] for QA verification.

Comment 9

[(Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo]

Attachment: Mockup for switching back to the old provider

Here's a mockup with some updated copy that tells people about the experiment. After seeing it, I absolutely agree that telling people about the study is the right thing to do.

Note that Google should only be restored if the user hasn't made any changes to the default search provider during the experiment.

Comment 10

[:Felipe Gomes (needinfo for replies!)]

Work done at https://hg.mozilla.org/users/felipc_gmail.com/search-experiment and to be reviewed through
bug 1029818

Comment 11

[Marco Mucci [:MarcoM]]

Hi Felipe, can you provide a point value.

Comment 12

[Marco Mucci [:MarcoM]]

Another question Felipe, should this bug be marked as [qa-] if the testing will be conducted through bug 1029818?

Comment 13

[:Felipe Gomes (needinfo for replies!)]

Setting p=3. Most of the testing will be done in bug 1029818, but since this is a tricky case, I think it's worth for us to leave [qa+] in this bug to verify the behavior described in comment 4.

Comment 14

[:Felipe Gomes (needinfo for replies!)]

The QA for everything related to the search experiment was done in bug 1029818 so I'll change what I said in comment 13 and mark this one as qa-

Comment 15

[Kamil Jozwiak [:kjozwiak]]

Marking this as verified as it was already tested in:
* bug #1029818 comment #34
* bug #1029818 comment #35

Used the following build:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b6/win32/en-US/ [Build ID: 20140630185627]

Comment 16

[Mark Banner (:standard8)]

Dropping NI from long-closed bug.