1717894 - Some about pages are blank until browser restart

Status: VERIFIED FIXED

(Firefox :: Normandy Client, defect)

Description

[Maria Heres, inactive, contact Ecosystem QA]

[Affected Versions]:

• Firefox Beta 90.0b11 (Build ID: 20210622185930)
• Firefox Nightly 91.0a1 (Build ID: 20210623095324)

[Affected Platforms]:

• Windows 10
• macOS 11.1
• Linux Mint 20

[Prerequisites]:

• Open an older version of Firefox Beta.

[Steps to reproduce]:

1. Open Hamburger menu > “Help” > “About Firefox” to trigger an update.
2. Wait for the update to download.
3. Click “Restart to Update Firefox”.
4. Dismiss the onboarding modal.
5. Navigate to “about:support” and observe the page.

[Expected result]:

• The page is populated.

[Actual result]:

• The various tables of the page are blank.

[Notes]:

• The issue seems reproducible when simply opening Firefox Nightly 91.0a1 (Build ID: 20210623095324) and navigating to "about:support" on Linux Mint 20.
• The issue is also reproducible for the "about:studies" page if the user is enrolled in an experiment before updating, though in that case it seems intermittent and seemed to occur less often for a multipreference default type recipe.
• The issue does not seem reproducible when updating Firefox Release 88.0.1 to 89.0.2 (Build ID: 20210622155641)
• Attached is a recording of the issue.

Comment 1

[Michael Cooper [:mythmon]]

I was able to reproduce this, and debugged it a bit. It seems like the promise ExperimenterManager.store.ready() is never resolving or rejecting. That sounds like a problem with Nimbus.

Andrei, do you know what might be happening here?

Comment 2

[Maria Heres, inactive, contact Ecosystem QA]

After further investigation, this is the result of mozregression:
Last good revision: 2c19c6b038a4f7741f97ae2fd1723e9e820547e6
First bad revision: bc5476061c43828c3fc868e216556f852df086e5
Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=2c19c6b038a4f7741f97ae2fd1723e9e820547e6&tochange=bc5476061c43828c3fc868e216556f852df086e5

Comment 3

[Michael Cooper [:mythmon]]

The commit Maria commented about is the cause of the apparent bug. However, after some investigation with Andrei, I believe that the root cause here is a race condition between UI startup and Normandy's initialization phase. Normandy relies on receiving notice that the UI has fully loaded before proceeding with initialization. In some cases this notice can come too early, before Normandy has set up the event listeners.

The blank about:support is due to the Nimbus machinery being more insistent on waiting for initialization, whereas Normandy will happily seem to work in a partially degraded state. However, I now think this bug is definitely a Normandy one.

Comment 4

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Encountered this issue when performing update tests to Firefox RC1 90.0 on beta and release-localtest channels. The about:support page contains no data after updating to 90.0.

Comment 5

[Michael Cooper [:mythmon]]

Attachment: Bug 1717894 - Register Normandy's UI ready observer before awaiting to avoid races r=andreio

Without this change, Normandy migrations could take longer to run than the UI took to start up, and so the UI ready notification would be sent before the listener was set up. This caused Normandy to sometimes never initialize, especially right after an upgrade.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/739435aeeb7e
Register Normandy's UI ready observer before awaiting to avoid races r=andreio

Comment 7

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/739435aeeb7e

Comment 8

[Maria Heres, inactive, contact Ecosystem QA]

I have confirmed that the issue seems to no longer be reproducible. The "about:support" page is no longer blank on startup on Linux Mint 20 and updating Nightly to the newest version didn't cause the "about:support" and "about:studies" pages to be blank on any of the tested OSs. Tested using Firefox Nightly 91.0a1 (Build ID: 20210708214616) on Linux Mint 20, macOS 11.1 and Windows 10.

Comment 9

[Julien Cristau [:jcristau]]

Wondering about the impact if normandy doesn't initialize in some cases; could this cause issues with studies during the 90 release cycle?

Comment 10

[Michael Cooper [:mythmon]]

The cases in which Normandy fails to initialize are well known: It's happens the first session after a Firefox upgrade with Normandy migrations. However, we often target those early session with experiments, and they are already rare since we throttle updates. Lets up lift this, if possible.

Comment 11

[Michael Cooper [:mythmon]]

Comment on attachment 9229970 [details]
Bug 1717894 - Register Normandy's UI ready observer before awaiting to avoid races r=andreio

Beta/Release Uplift Approval Request

• User impact if declined: about:support may be blank the first session after upgrading to 90.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The change to Normandy's initialization is small, well understood, and tested.
• String changes made/needed: none

Comment 12

[Julien Cristau [:jcristau]]

Comment on attachment 9229970 [details]
Bug 1717894 - Register Normandy's UI ready observer before awaiting to avoid races r=andreio

It's too late for 90.0 (ships tomorrow); I'll consider this for a dot release.

Comment 13

[Michael Cooper [:mythmon]]

Ah, right. I hadn't realized where we were in the process. I don't think this should be a dot release driver, but would be fine as a ride along. Note that the 90.0 -> 90.0.1 upgrade won't be affected by this, since it doesn't contain any Normandy migrations. Would you like me to file a release uplift request?

Comment 14

[Julien Cristau [:jcristau]]

It's fine, I switched the request from beta to release :)
I imagine there's still some benefit in fixing this for the 89 → 90.0.1 case.

Comment 15

[Julien Cristau [:jcristau]]

Comment on attachment 9229970 [details]
Bug 1717894 - Register Normandy's UI ready observer before awaiting to avoid races r=andreio

approved for 90.0.1

Comment 16

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-release/rev/4fa9f72d13b6

Comment 17

[Maria Heres, inactive, contact Ecosystem QA]

I have verified that the issue seems to no longer be reproducible on the Firefox 90.0.1 candidate (Build ID: 20210716144314). The "about:support" page does not remain blank on Linux Mint 20 and enrolling in a study and restarting the browser does not seem to blank the "about:studies" page on any OS tested. I could not test this using the update scenario. Tested on Windows 10, Linux Mint 20 and macOS 11.1.

Comment 18

[Julien Cristau [:jcristau]]

Added to 90.0.1 release notes: "Fixed a race on startup that caused about:support to end up empty after upgrade"