1859437 - It's possible for a "new user" to see a what's new page followed by a moments page on the next restart

Status: NEW

(Firefox :: Messaging System, defect, P3)

Description

[Dan Mosedale (:dmosedale, :dmose)]

1. Get 117.0.1 https://support.mozilla.org/en-US/kb/install-older-version-firefox
2. Use it to create a new profile using about:profiles
3. Copy in user.js with test id for natural enrollment in the new user experiment
4. Quit
5. Start with the new profile
6. "Check for updates" to get 118 downloaded, but DO NOT RESTART
7. Check about:studies to see that natural enrollment has occurred
8. Search about:config for 'startup', make sure browser.startup.homepage_override.once pref does NOT EXIST or is NOT SET
9. Restart
10. Make sure 118 WNP seen
11. Search about:config for 'startup', make sure the browser.startup.homepage_override.once pref EXISTS and IS SET to the WNP homepage
12. Restart
13. Check that Moments page is seen

Broadly, we'd like to figure out how much of an edge case this is. To help with that, some open questions that would help us assess that include

1. this should presumably affect 118 to 119 updates as well. Does it?
2. do the "search about:config steps" ever fail to return what's written above? If/when they do, does it affect what gets shown after the next restart?
3. current hypothesis is that if the browser stays open for 5 minutes after step 7, and then one reloads about:config, that pref should then be set. Is this correct?

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

Open question 4:

4. Because these experiments are not sticky, if the test in comment 0 is timed in such a way that the user starts with a 27-day-old profile (or so times.json says) and a system clock that's a small number of minutes before midnight, and the user gets unenrolled from the new users experiment and then enrolled into the existing users experiment, can we confirm that the message blocking prevents the message from being shown twice? It should...

Note that these open questions are in order of importance. Which is to say that 4 feels likely to be extra edge-casey, so I'm not very concerned about it. But if, say, :aminomancer were to tell me that I'm wrong, or that I hadn't thought of some variant that is less edge-casey, now would be a good time to hear that. So I'm going to needinfo him... :-)

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

QA looked into this last night, and wrote:

I have tested the scenario from the description and I can confirm that only the WNP page is displayed after a browser even if the “MOMENTS_PAGE_SET event” was sent before updating the browser. The VPN Moments page will be displayed at the next browser restart after that.

1. this should presumably affect 118 to 119 updates as well. Does it?

The same scenario as above applies when updating from Firefox Beta 118.0b9 to Firefox Beta 119.0 (used force enroll for this scenario).

2. do the "search about:config steps" ever fail to return what's written above? If/when they do, does it affect what gets shown after the next restart?

3. current hypothesis is that if the browser stays open for 5 minutes after step 7, and then one reloads about:config, that pref should then be set. Is this correct?

Regarding the “browser.startup.homepage_override.once” pref. For the vast majority of times when we have verified this scenario, the pref was set immediately after the browser was enrolled in the experiment. However, we have not observed any difference between the times when the pref was set immediately after enrollment and the times when it was set after approximately 5 minutes. The WNP was always triggered first and the VPN Moments page after the following browser restart.

4. Because these experiments are not sticky, if the test in comment 0 is timed in such a way that the user starts with a 27-day-old profile (or so times.json says) and a system clock that's a small number of minutes before midnight, and the user gets unenrolled from the new users experiment and then enrolled into the existing users experiment, can we confirm that the message blocking prevents the message from being shown twice? It should...

We have also tested the scenario where the targeting of the “New Users” experiment is invalidated, the client is unenrolled from it, and enrolled in the “Existing Users” one. In this case, we can confirm that the VPN Moments page is only triggered once generally during the New Users experiment.

We have tested the above scenarios using Firefox Release 117.0.1 (Build ID - 20230912013654), the latest Firefox Release 118.0.2 (Build ID - 20231009140911), and Firefox Beta 118.0b9 (Build ID - 20230914180032) installed on Windows 10 x64 and macOS 13.5.1.

I hope this will help. Also, if you have any other questions or do you want us to cover any additional scenarios, please don’t hesitate to ping me.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:lsmith, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[Shane Hughes [:aminomancer]]

NI Hanna since you have more context about the WNP.