1711206 - The "expPrefChanged" events are not sent when changing the experiment prefs after browser update

Status: VERIFIED FIXED

(Firefox :: Normandy Client, defect)

Description

[Simona Rosu [:srosu], Ecosystem QA, PTO - NI to: [e-mail]]

Attachment: prefchanged_normandy_update.gif

[Affected versions]:

• Firefox Nightly 90.0a1 (Build ID: 20210513214800)

[Affected Platforms]:

• Windows 10 x64
• Linux Mint 20
• MacOS 11.2.3

[Prerequisites]:

• Have an older Firefox Nightly browser installed. (e.g. 89)
• Have a Normandy recipe for a single preference experiment set up and published.
• Have a Firefox profile that is enrolled in the experiment.

[Steps to reproduce]:

1. Open the Firefox profile from prerequisites and update it. ("Hamburger menu" button>Help>About Firefox>"Restart to update" button)
2. Navigate to the “about:config” page and confirm that you understand the consequences.
3. Search for the experiment set prefs and change each experiment prefs to different values.
4. Navigate to the about:telemetry#events-tab page and search for the "expPrefChanged" event.

[Expected result]:

• An "expPrefChanged" event was sent for every preference change.

[Actual result]:

• No "expPrefChanged" event is registered.

[Notes]:

• This issue is only reproducible with default branch-type preferences.
• Used the following stage recipe: 1388.
• Attached a screen recording of the issue and the “shield-preference-experiments.json” file.

Comment 1

[Simona Rosu [:srosu], Ecosystem QA, PTO - NI to: [e-mail]]

Attachment: shield-preference-experiments.json

Comment 2

[Michael Cooper [:mythmon]]

Where in your steps do you change from version 89 to 90?

Comment 3

[Michael Cooper [:mythmon]]

In the provided JSON, the preference is listed as being overridden. This sounds like Normandy correctly detected the change, and either didn't send the telemetry, or the telemetry wasn't in the expected place.

I think I saw the browser console in the recording, but it was entirely blank. I'd expect there to be some activity there, even if it wasn't Normandy-related. Had it not loaded yet? It is possible that Normandy is sending an error related to the format of the Telemetry ping.

Comment 4

[Simona Rosu [:srosu], Ecosystem QA, PTO - NI to: [e-mail]]

Hi @Mythmont, for comment 2 the browser update is made in step 1, after you are enrolled in the study.
For comment 3 the browser console was entirely blank because I’ve tested this scenario using the “XPCSHELL_TEST_PROFILE_DIR=foo” variable. However, I’ve just retested this scenario without using the variable and I didn’t encounter any kind of errors in the browser console after the prefs changes.

Comment 5

[Michael Cooper [:mythmon]]

Attachment: Bug 1711206 - Apply Normandy startup preferences before Normandy migrations r=gijs

This is a major change to the sequencing of Normandy's startup, since it now
applies startup preferences before the first async. This makes preference
changes happen synchronously during BrowserGlue's initialization phase. This
timing is something we have worked around frequently in the past.

This may affect some other features, but I believe they will all still work as
expected. Future features will be easier to integrate with Normandy, since the
timing of this phase is more predictable now.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fd68113fecb1
Apply Normandy startup preferences before Normandy migrations r=Gijs

Comment 7

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/fd68113fecb1

Comment 8

[Simona Rosu [:srosu], Ecosystem QA, PTO - NI to: [e-mail]]

I have verified this issue using the latest Nightly 90.0a1 (Build ID: 20210526211756) on Windows 10 x64, Ubuntu 20.04 and macOS 11.2.3.

• The "expPrefChanged" events are sent when changing the experiment prefs after browser update.