1038493 - Translation experiment should wait 5 days before activating treatment group

Status: RESOLVED WONTFIX

(Firefox :: Translations, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

On installation, the experiment will pick (by random 50% chance) users for the treatment or control groups, and store this info in prefs+experiment branch Language detection will be activated, but the UI remain inactive.

After 10 wall days, users who were chosen to be part of the treatment group will get the showUI pref flipped, and then will start seeing the translation feature.

I'm storing the branch info both in prefs+experiment as a way to test the hypothesis for bug 1038174. This experiment goes with the pref as the official source, and we'll be able to see in the FHR data if the branch assignment didn't reach 100% of installs, which means it was probably set but not saved for some users.

Comment 1

[:Felipe Gomes (needinfo for replies!)]

Attachment: Diff from previous version

Comment 2

[:Felipe Gomes (needinfo for replies!)]

Attachment: Final bootstrap.js (templated)

Comment 3

[Marco Mucci [:MarcoM]]

Hi Felipe, can you provide a point value and if the bug should be marked as [qa+] or [qa-]

Comment 4

[:Felipe Gomes (needinfo for replies!)]

qa- as QA should be done in bug 1035333

Comment 5

[:Felipe Gomes (needinfo for replies!)]

Attachment: Diff from previous experiment's bootstrap.js

Now with more logging through TelemetryLog

Comment 6

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8455889 [details] [diff] [review]
Diff from previous experiment's bootstrap.js

Now with more logging through TelemetryLog

Comment 7

[:Felipe Gomes (needinfo for replies!)]

Attachment: Final bootstrap.js (templated)

Comment 8

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8455889 [details] [diff] [review]
Diff from previous experiment's bootstrap.js

(Sry for the bugspam.. it's late and I set the obsolete flag in the wrong patch)


One thing to note is that it would be interesting to know (and the logs would tell) if the pref file is not being saved properly for some users. But the usage of savePrefFile which I added here will likely reduce a lot the chance of that happening in comparison with the two previous experiments.

It would be an exchange of ensuring this doesn't happen to users of this experiment, vs. testing the theory (and figuring out what % of users this affects).

Comment 9

[Benjamin Smedberg]

Comment on attachment 8455889 [details] [diff] [review]
Diff from previous experiment's bootstrap.js

Are there no available constants for reason 7 (upgrade)? Sounds like it should at least be a constant at the top of this file.

Why is pref_install_time a char pref that you then convert back into an int (instead of just being an int pref consistently)? Does that have something to do with 64-bit values?

I believe the daysElapsed >= 10 case should also have a savePrefFile call.

r=me with those changes or explanations.

Comment 10

[:Gavin Sharp [email: [e-mail]]]

I'm worried about the complication this introduces (not being entirely familiar with the branch code), and Chad's worried about the implications on the amount of data we collect (and time before we collect it). Do we really want this?

Comment 11

[:Gavin Sharp [email: [e-mail]]]

(In reply to Benjamin Smedberg  [:bsmedberg] Away 19-July through 3-Aug from comment #9)
> Are there no available constants for reason 7 (upgrade)?

The reason constants from https://developer.mozilla.org/en-US/Add-ons/Bootstrapped_extensions#Reason_constants (e.g. ADDON_UPGRADE) are defined in all bootstrap.js scopes IIRC.

Comment 12

[:Felipe Gomes (needinfo for replies!)]

(In reply to :Gavin Sharp [email: [e-mail]] from comment #10)
> I'm worried about the complication this introduces (not being entirely
> familiar with the branch code), and Chad's worried about the implications on
> the amount of data we collect (and time before we collect it). Do we really
> want this?

This was suggested by Saptarshi on his doc analyzing the initial data from the current experiment. I think him/Chad should talk to see how much we want this. I think the delay could be lowered to 5 days though, 10 days sounds like a lot.

The branch is not really used for anything here other than helping the data report and let us understand better if the current branch code is unreliable.

But it does add the complication of the pref setting, and the update path in case we need to use it.

(In reply to Benjamin Smedberg  [:bsmedberg] Away 19-July through 3-Aug from comment #9)
> Why is pref_install_time a char pref that you then convert back into an int
> (instead of just being an int pref consistently)? Does that have something
> to do with 64-bit values?

I tried and an int pref can't properly store the value from Date.now()

Comment 13

["Saptarshi Guha[:joy]"]

Ye(In reply to :Felipe Gomes from comment #12)

> This was suggested by Saptarshi on his doc analyzing the initial data from
> the current experiment. I think him/Chad should talk to see how much we want
> this. I think the delay could be lowered to 5 days though, 10 days sounds
> like a lot.
> 
Yes i guess 5 days is okay. 10 days is not too much given that users dont use FF every day.
When you say 5 days do you mean 5 days of use? or 5 calendar days?

I understand we want more time for the actual UI to be shown. So 5 (ideally 7) calendar days is okay.

Comment 14

[:Felipe Gomes (needinfo for replies!)]

The code is going by calendar days, not days of use.

Comment 15

[:Felipe Gomes (needinfo for replies!)]

Since most users don't use the browser daily, I made it 5 days because that will lead the interval to naturally fall between 5 - 8 days.

http://hg.mozilla.org/webtools/telemetry-experiment-server/rev/a3c4c7e073a9

Comment 16

[:Felipe Gomes (needinfo for replies!)]

Backed out in http://hg.mozilla.org/webtools/telemetry-experiment-server/rev/03a66c01de63 because it was decided not to go in this direction.