1553198 - Preference Experiments are expired in the same session they are fetched with “app.normandy.dev_mode” set to true

Status: VERIFIED FIXED

(Firefox :: Normandy Client, defect, P1)

Description

[Andreea Cupsa [:acupsa], Experiments QA]

Attachment: 2019-05-21 18_16_32-Window.png

[Affected versions]:

• Firefox Beta 68.0b3 [Build ID 20190520131124]
• Firefox Nightly 69.0b1 [Build ID 20190521094447]

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Access to NORMANDY Delivery Console https://delivery-console.stage.mozaws.net/ is needed.
• Set the app.normandy.dev_mode preference to true to run recipes immediately on startup.
• Set the app.normandy.logging.level preference to 0 to enable more logging.
• Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
• Set the preference value for app.normandy.api_url set to https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1.
• Set the services.settings.server preference to https://settings.stage.mozaws.net/v1.
• An preference experiment recipe was already created and is live #720
• Have the browser console opened.

[Steps to reproduce]:

1. Open the browser with the profile targeted in the prerequisites.
2. Wait ~2-5 minutes and restart the browser.
3. Search in the browser console for the recipe from the prerequisites and observe the logs.

[Expected result]:

• The recipe from the prerequisites is fetched, executed and remains active.

[Actual result]:

• The recipe from the prerequisite is fetched, executed and stopped.

[Notes]:

• This issue is not reproducible on Firefox Release 67 [Build ID 20190516215225,

User Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0]

• I used the filter expression: 'andreea'|preferenceExists and the following recipes: #720, #716
• Here is a screen recording with the issue.
• Attached a screenshot of the logs from the browser console.

Comment 1

[Michael Cooper [:mythmon]]

I noticed the same errors from bug 1553125. I wonder if these errors cause actions to still run, causing unenrollment.

Can you check the telemetry events associated with this test? There should be an unenrollment event that includes some specific details about the unenrollment.

Comment 2

[Andreea Cupsa [:acupsa], Experiments QA]

Attachment: Normandy logs and telemetry events.txt

I am attaching a file with the browser console logs and the telemetry unenroll events. It seems like they are unenrolled as not seen. However, this does not make sense to me considering that the recipes are still active.

Comment 3

[Michael Cooper [:mythmon]]

Andreea: Thanks for the logs. It is indeed weird that we're seeing "recipe-not-seen" and also that it seems like all the unenrollments are doubled. That's concerning. I also notice that this is using a recently landed change to preference experiments we did in bug 1547034. Can you see if this is reproduceable on a build that doesn't include that change?

Ethan: Since this looks related to multi prefs, can you take a look as well?

Comment 4

[Andreea Cupsa [:acupsa], Experiments QA]

This issue is not reproducible with a previous build which does not contain the change.
I also ran a regression range and I got the following pushlog and looks like you were correct and bug 1547034 caused this.

Comment 5

[Ethan Glasser-Camp (:glasserc)]

Attachment: Bug 1553198: Introduce PreferenceExperiments storage format v3

This format includes an "actionName" for each experiment which
identifies the source of the experiment. This makes it possible for
each experiment type to identify which experiments it should clean up
vs. which it should leave alone because they don't belong to it.

Comment 6

[Ethan Glasser-Camp (:glasserc)]

Attachment: Bug 1553198: PreferenceExperiments.start takes an actionName

Comment 7

[Ethan Glasser-Camp (:glasserc)]

Attachment: Bug 1553198: PreferenceExperiment only cleans up its own experiments

Comment 8

[Ethan Glasser-Camp (:glasserc)]

Nice catch. Thanks for looping me in :mythmon!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=19c876c17927757447057e9b622ecf18ed109bf2

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/81d97da2c8b0
Introduce PreferenceExperiments storage format v3 r=mythmon
https://hg.mozilla.org/integration/autoland/rev/bb245437de64
PreferenceExperiments.start takes an actionName r=mythmon
https://hg.mozilla.org/integration/autoland/rev/2e6402ff4bf6
PreferenceExperiment only cleans up its own experiments r=mythmon

Comment 10

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/81d97da2c8b0
https://hg.mozilla.org/mozilla-central/rev/bb245437de64
https://hg.mozilla.org/mozilla-central/rev/2e6402ff4bf6

Comment 11

[Ethan Glasser-Camp (:glasserc)]

:acupsa, can you please verify that this is fixed in Nightly when you get a chance? I'll request uplift as soon as that's done.

Comment 12

[Andreea Cupsa [:acupsa], Experiments QA]

This issue is no longer reproducible on Firefox Nightly 69.0a1 and recipes are no longer expired in the same session they are fetched.

However, due to the fact that fetching naturally recipes are blocked by Bug 1553125, I have verified this issue on Firefox Nightly 69.0a1 (Build ID 20190526211422) downloaded with mozregression and Nightly 69.0a1 (Build ID 20190526211422) from ftp.mozilla.org and forced "fetching" the recipes with the remote-settings-devtools addon.

Comment 13

[Ethan Glasser-Camp (:glasserc)]

Comment on attachment 9067086 [details]
Bug 1553198: PreferenceExperiment only cleans up its own experiments

Beta/Release Uplift Approval Request

• User impact if declined: Preference experiments on Normandy will not work properly. This hinders Mozilla's ability to perform experiments to inform product direction.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a targeted fix with no architectural changes.

Another alternative is to backout bug 1547034 on beta and let this only ride for 69. This is probably the safest thing but having this land in ESR would reduce Mozilla's maintenance burden by a lot.

• String changes made/needed: none

Comment 14

[Julien Cristau [:jcristau] (back April 22)]

Michael what's your take here on uplift vs backout?

Comment 15

[Michael Cooper [:mythmon]]

As Ethan pointed out, keeping these patches is slightly risky. The risk is that if there are other bugs we haven't yet found, then preference experiments (both the new multi-value ones and the older single-value ones) may be impacted. Preference experiments are the preferred way of running experiments. On the other hand, multi-value pref experiments have been a much requested feature for a long time, and I would be very happy to have them in 68. I've only seen this one bug report, though QA hasn't done a dedicated set of tests on this new feature.

I don't think that ESR plays much into this debate, since we rarely target ESR with preference experiments. The ambient QA coverage we have through Normandy's general smoke tests is enough to give me confidence that this feature is safe to land in 68, with the patch in this bug uplifted.

I think we should uplift this patch. If we run into further problems, I'd favor backing out both patches at that point.

Comment 16

[Julien Cristau [:jcristau] (back April 22)]

Comment on attachment 9067086 [details]
Bug 1553198: PreferenceExperiment only cleans up its own experiments

Thanks both! Let's get this in 68.0b6.

Comment 17

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/releases/mozilla-beta/rev/49ab063b752d
https://hg.mozilla.org/releases/mozilla-beta/rev/bc0971ecb6e6
https://hg.mozilla.org/releases/mozilla-beta/rev/2841c37353a5

Comment 18

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Ethan, can you check if too many patches got uplifted to beta, please?

Comment 19

[Ethan Glasser-Camp (:glasserc)]

Hi, I didn't understand the question. To me it seems like the right number of patches got uplifted, but I didn't uplift them myself and apart from the links posted to them here, I haven't really looked at the mozilla-beta branch to see what else might have been uplifted. Could you please elaborate on your question?

Comment 20

[Julien Cristau [:jcristau] (back April 22)]

You requested uplift on just one of the patches. All three got uplifted. That may or may not have been the right thing to do.

Comment 21

[Ethan Glasser-Camp (:glasserc)]

I requested uplift on the one patch, plus checked the box for "Also include the other two patches". I intended for all three to be included.

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Only one of the 3 patches has approval-mozilla-beta+, that's the reason I wondered if all 3 were supposed to get uplifted.

Comment 23

[Ciprian Muresan [:cmuresan], Ecosystem QA]

I have verified that the issue is no longer reproducible on the latest Firefox Beta (68.0b8, Build ID 20190606101422) on Windows 10, macOS 10.14, and Arch Linux 14.4.3.