1749758 - [Intermittent] The Data Collection option is re-enabled after previously disabling it when opting-in in the experiment

Status: VERIFIED FIXED

(Firefox :: Address Bar, defect, P1)

Description

[Carmen Fat [:cfat] - Ecosystem QA, PTO until July 4th - NI to: [e-mail]]

[Affected Versions]:

• Firefox Beta 97.0b2 (Build ID: 20220111185943)
• Firefox Nightly 98.a1 (Build ID: 20220111093827)

[Affected Platforms]:

• Windows 10 x64
• Ubuntu 20.04 x64
• macOS 10.15.7

[Prerequisites]:

• Have Firefox Beta 97.0b2 downloaded on your computer.
• Have the "browser.search.region" set to "US".
• Have one of the treatment user.js on your computer.
• Make sure there is no other modal displayed when starting the browser (browser default, onboarding for new users etc).

[Steps to reproduce]:

1. Open Firefox Beta 97.0b2.
2. Navigate to the “about:support” page and paste the user.js file into the Profile folder.
3. Restart the browser.
4. Click the “Find out how” button from the modal.
5. Select the “Allow” option and click the “Save preferences” button.
6. Open the "about:preferences” page and verify that the 3rd toggle from the Firefox Suggest section is enabled.
7. Go back to “about:config" and set as false the browser.urlbar.quicksuggest.dataCollection.enabled preference.
8. Restart the browser.
9. Select the "about:preferences” tab and observe the 3rd toggle from the Firefox Suggest section.

[Expected results]:

• The 3rd toggle is disabled.

[Actual results]:

• The 3rd toggle is enabled.

[Notes]:

• Here is a screen recording of the issue (captured after opting-in the experiment).
• The behavior is reproducible only once. As seen in the video, I have set the preference to false again and followed the remaining steps, but the issue was no longer reproducible (the value remained set to false as expected.

Comment 1

[Drew Willcoxon :adw]

Thanks Carmen, out of all the bugs you filed today, this one worries me the most. Could you answer some questions please?

• Please try reproducing with the preferences below. No need to enroll or use the modal, I'm only interested in whether the prefs have the wrong value after you toggle them on about:config and then restart.
  • browser.altClickSave
  • browser.urlbar.suggest.quicksuggest.nonsponsored
  • devtools.console.stdout.content
• If you reduce the STR to only toggle the pref on about:config without enrolling or using the modal, does it still happen?
• How often does it happen, how hard is it to reproduce? You said it's reproducible only once, but I'm not sure if you mean it happened only one time overall or it happens only once per profile.
• It looks like you are restarting by using the key shortcut in the browser console, correct? Does it happen if you restart in a way a user typically would?
• Are you using a slow machine/VM?

Right now my guess is it's related to how quickly you restart after toggling the pref and/or the way you restart, and it should be reproducible with other prefs, but I don't have a good explanation.

Comment 2

[Carmen Fat [:cfat] - Ecosystem QA, PTO until July 4th - NI to: [e-mail]]

1. We verified the 3 preferences from the above comment using several Firefox profiles and every time we restarted Firefox, the preferences values remained the same as we set them, as expected.

2. I managed to reproduce the issue only one time without enrolling in the experiment or using the opt-in modal. Here are the STR:

• set the browser.search.region to US;
• restart the browser;
• navigate to about:config and set the Data Collection pref (browser.urlbar.quicksuggest.dataCollection.enabled) to true;
• open the about:preferences page in a new tab and verify that the 3rd toggle (Improve the Fx Suggest experience) is enabled;
• restart the browser;
• go to about:config and set the Data Collection pref to false;
• go to the about:preferences page and verify that the 3rd toggle is false.
  At the last step, the 3rd toggle was enabled, so I went back to about:config and noticed that the value was switched back to true automatically.

3. It happens very rarely and today we reproduced it on 2 machines only.

• Yesterday I was able to reproduce it 4 times, today I was able to reproduce it only one time, using the scenario described at point 2 (using a Windows 10 machine).
• The other machine (Ubuntu 20.04 x64) also reproduces it very rarely.
• As for my statement that it's reproducible only once, I meant that the behavior reproduces only once per affected profile. In the screen recording from the Description, the issue can be observed starting with second 5 until second 22. After the Data Collection preference was automatically set back to true, I tried to reproduce the issue again by setting it to false, you can see this in the video starting with second 25. However, this time, the preference remained set to false, as it should have happened from the beginning.

4. Yes, the scenario from the Description is reproduced when the browser is closed via the “X” button as well.

5. None of the machines use VM. One of the machines is indeed one of the slow Ubuntus 20.04 x64 where we previously encountered the issues related to low-performance. The other machine is a Windows 10 x64 which performs better compared to the Ubuntu one (on the Windows machine I wasn’t able to reproduce the performance issues we previously found on Ubuntu).

We will continue with exploratory testing on this behavior and update the bug if we find new details.

Comment 3

[Drew Willcoxon :adw]

Thanks Carmen. I don't know what the problem could be.

I made some builds with extra logging. Could you please reproduce it on one of them, and after the problem happens open the browser console, filter on messages with XXXadw in them, and paste the output here? No need to do this more than once. Please use whichever build/platform is easiest for you.

• Windows
• Mac
• Linux

Treeherder link

After that, please try the following as time permits:

• Instead of toggling the pref on about:config, please try to reproduce it in the two ways a user typically would: (1) by opting out of the modal, (2) by opting in and then turning off the toggle switch on about:preferences.
• Please try again to reproduce it with those other three prefs I mentioned. I really want to determine if this is a problem with the dataCollection.enabled pref in particular or all prefs.

Comment 4

[Carmen Fat [:cfat] - Ecosystem QA, PTO until July 4th - NI to: [e-mail]]

Thank you for the try-builds.

• We managed to reproduce the issue on multiple machines. Here are the logs that appeared in the Browser Console after reproducing the issue:

***XXXadw UP._updateFirefoxSuggestScenarioHelper, scenario= online isStartup= true default quicksuggest.dataCollection.enabled= false UrlbarPrefs.jsm:778:13 ***XXXadw UP._ensureFirefoxSuggestPrefsMigrated, currentVersion= 2 lastSeenVersion= 2 UrlbarPrefs.jsm:874:13 ***XXXadw UP.observe, quicksuggest.dataCollection.enabled= true UrlbarPrefs.jsm:1146:15 ***XXXadw observe@resource:///modules/UrlbarPrefs.jsm:1152:34 set valueFromPreferences@chrome://global/content/preferencesBindings.js:634:26 set value@chrome://global/content/preferencesBindings.js:535:11 userChangedValue@chrome://global/content/preferencesBindings.js:210:11 onInput@chrome://global/content/preferencesBindings.js:245:12 handleEvent@chrome://global/content/preferencesBindings.js:288:23 UrlbarPrefs.jsm:1152:15

• After comparing the logs from my machine with the logs of a different machine, I noticed a small difference between the logs. More specifically, several strings from my logs use upper case letters, while the log from the other machine contains strings written in lowercase letters. Not sure how relevant is this difference though (please see the attachment with the logs comparison).

• On profiles we didn’t manage to reproduce the issue, we only received the following logs:

***XXXadw UP._updateFirefoxSuggestScenarioHelper, scenario= online isStartup= true default quicksuggest.dataCollection.enabled= false UrlbarPrefs.jsm:778:13 ***XXXadw UP._ensureFirefoxSuggestPrefsMigrated, currentVersion= 2 lastSeenVersion= 2 UrlbarPrefs.jsm:874:13

• As for the last scenarios, we were not able to reproduce the issue in the ways a normal user would do. Also, the values of the 3 prefs mentioned in comment 1 remain the same after restarting the browser. We tried these scenarios on several different profiles using multiple machines, but without luck.

Comment 5

[Carmen Fat [:cfat] - Ecosystem QA, PTO until July 4th - NI to: [e-mail]]

Attachment: bug logs.jpg

Comment 6

[Drew Willcoxon :adw]

Attachment: lldb log

Thanks, that's very helpful and it allowed me to reproduce it. The problem is the toggle switch states are being stored in session store. Intermittently, when the page is loaded due to session restore, the restored state of each toggle can override the value of the underlying pref. So, this bug can happen in two cases: (1) after you opt-in using the modal, (2) after you change the pref value using about:config. In both cases, the state of the toggle in session store may not be the same as the value of the underlying pref.

This bug can happen for all three of the Suggest toggles.

In the real world, I don't think we need to worry about users changing the value in about:config. That leaves the following possible scenarios where this bug can happen, in both cases for the data-collection pref and third toggle:

1. The user enables session store, opens about:preferences#privacy, leaves the third toggle off (or turns it on then off), opts IN using the modal, restarts Firefox, and clicks the about:preferences tab. In this case, the toggle and pref may be intermittently turned off.
2. The user enables session store, opens about:preferences#privacy, turns the third toggle on, opts OUT using the modal, restarts Firefox, and clicks the about:preferences tab. In this case, the toggle and pref may be intermittently turned on.

I've attached an lldb log that shows both the JS and C++ call stacks at the time the data-collection pref is set by session store. The JS stack isn't interesting because all it contains is a single call to Preferences.handleEvent() in preferencesBindings.js. All the session store frames are in the C++ stack. (I added a call to Math.sin() in Preferences.handleEvent() to let me break in the C++ by breaking on its C++ implementation, js::math_sin().)

Comment 7

[Drew Willcoxon :adw]

[Tracking Requested - why for this release]:
This is a blocker for the Firefox Suggest experiment in a 96 dot release.

Comment 8

[Drew Willcoxon :adw]

Attachment: Bug 1749758 - Disable session store for the Firefox Suggest toggles/checkboxes in about:preferences.

Please see bug 1749758 comment 6 for background. In summary, session store is
restoring the states of the Firefox Suggest <html:input> checkboxes in
about:preferences#privacy, which intermittently happens after their states
are initialized by the preferences code. If a checkbox's session store state
does not match the value of its underlying pref, then the checkbox and pref both
end up with the wrong value because session store sets the wrong checked state
of the checkbox, which fires an input event, which then updates the pref.

This can only happen when the pref value is changed outside of
about:preferences, which is indeed the case for the Firefox Suggest pref
discussed in the bug.

To fix it, I tried adding autocomplete=off to each input, but unfortunately
that does not work because input[type=checkbox] is specifically not allowed to
opt out of autocomplete/session store. The code path for restoring input
checkboxes is:

[snipped to avoid annoying effects of Phabricator Markdown in Bugzilla]

So, the only way to disable autocomplete seems to be to associate the input with
a form and set autocomplete=off on the form, so that's what I've done.

It would be simpler to make these XUL checkboxes instead, which AFAICT don't
participate in session store, but these checkboxes need the toggle-switch
styling.

Finally, this is a problem for all <html:input>s used in about:preferences. I
didn't update any others because I want to keep this scoped for uplift, and
maybe the other input checkboxes should be XUL checkboxes anyway?

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/369f00fc613e
Disable session store for the Firefox Suggest toggles/checkboxes in about:preferences. r=Gijs,preferences-reviewers

Comment 10

[Drew Willcoxon :adw]

Comment on attachment 9259214 [details]
Bug 1749758 - Disable session store for the Firefox Suggest toggles/checkboxes in about:preferences.

Beta/Release Uplift Approval Request

• User impact if declined: This is a blocker for the Firefox Suggest experiment in a 96 dot release.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Should be clear from the bug comments
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a very small patch that only adds a hidden <form> element to the Firefox Suggest preferences UI on about:preferences.
• String changes made/needed:

Comment 11

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/369f00fc613e

Comment 12

[Carmen Fat [:cfat] - Ecosystem QA, PTO until July 4th - NI to: [e-mail]]

• We have verified this issue on the latest Nightly 98.0a1 (Build ID: 20220116095124) and the Firefox 96.0.2 try-build (Build ID: 20220115220116), on Windows 10 x64, macOS 10.15.7, and Ubuntu 20 x64.

• The Data Collection option is no longer re-enabled and honors the user's choice. We tested the scenario from the Description on 3 different machines we have previously reproduced the issue, using a total of 60 new Firefox profiles.

Comment 13

[(PTO June 19-July 7) Ryan VanderMeulen]

Comment on attachment 9259214 [details]
Bug 1749758 - Disable session store for the Firefox Suggest toggles/checkboxes in about:preferences.

Approved for 97.0b5.

Comment 14

[(PTO June 19-July 7) Ryan VanderMeulen]

https://hg.mozilla.org/releases/mozilla-beta/rev/da18515043ce

Comment 15

[Dianna Smith [:diannaS]]

Comment on attachment 9259214 [details]
Bug 1749758 - Disable session store for the Firefox Suggest toggles/checkboxes in about:preferences.

Approved for 96.0.2

Comment 16

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-release/rev/d6c785be4548

Comment 17

[Carmen Fat [:cfat] - Ecosystem QA, PTO until July 4th - NI to: [e-mail]]

• We have verified this issue on the latest Beta 97.0b5 (Build ID: 20220118185733) and the latest Firefox 96.0.2 (Build ID: 20220119040736), on Windows 10 x64, macOS 10.15.7, and Ubuntu 20 x64.

• The Data Collection option is no longer re-enabled and honors the user's choice. We tested the scenario from the Description on different machines we have previously reproduced the issue, using a total of 40 new Firefox profiles.