1626904 - [Experiment] Clients enrolled in the Do No Harm experiment could end up not seeing the new about:welcome page

Status: RESOLVED WORKSFORME

(Firefox :: Messaging System, defect)

Description

[Ciprian Muresan [:cmuresan], Ecosystem QA]

Attachment: firstrun-enrollment-fail.gif

[Affected Platforms]:

• All Windows

[Affected Versions]:

• Firefox Beta 75.0, Build ID 20200331175109

[Prerequisites]:

• Have an active multipref recipe.
• Have the recipe set-up in such a way that they target your profile. (this is the recipe I used)
• Download this prefs.js file on your computer.

[Steps to reproduce]:

1. Open the Browser with the Profile chooser.
2. Create a new profile but DO NOT open it.
3. Navigate to the local profile directory.
4. Place a copy of the prefs.js file from the prerequisites in the profile folder.
5. Open the Profile using Firefox Beta 75.0 and observe the behavior.

[Expected result]:

• The simplified about:welcome page is displayed.

[Actual result]:

• The default about:welcome page is displayed.

[Notes]:

• One of two things can happen in this case:
  • the client is NOT enrolled in the experiment because Normandy was not fast enough when executing the recipe
    OR
  • the client IS enrolled in the experiment, but the default page is still displayed because it loaded faster.
• We've also set-up a recipe in the Prod environment of Normandy and the issue is reproducible there as well.
• We've also used attribution code and the stub installer method of starting up Firefox for the first time and the issue is reproducible in this way as well. Normandy is either too slow in enrolling or the page is too fast when it is displayed.
• I have attached a screen recording of the behavior.

Comment 1

[Punam Dahiya [:pdahiya]]

Hi Marius

Looking at prefs.js file, it has user_pref("trailhead.firstrun.didSeeAboutWelcome", true). For a new profile trailhead.firstrun.didSeeAboutWelcome shouldn't exist . I am not able to view the recipe but I know from previous releases first run onboarding experiments, users are not enrolled if trailhead.firstrun.didSeeAboutWelcome is true. Hope it helps. Thanks!

Comment 2

[Punam Dahiya [:pdahiya]]

NI mythmon to help debug the issue and inputs. Thanks

Comment 3

[Michael Cooper [:mythmon]]

The recipe contains a clause

(
  !("trailhead.firstrun.didSeeAboutWelcome"|preferenceValue)
  || normandy.studies.pref[normandy.recipe.arguments.slug]
)

This is intended to prevent enrollment if the preference trailhead.firstrun.didSeeAboutWelcome is set to true. As Punam points out, you have that preference set to true, which means that you are incompatible with the recipe. The system is behaving as expected.

For those without access to Delivery Console, this is the public API version of the recipe revision that was used: https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v3/recipe_revision/3455/

Comment 4

[Ciprian Muresan [:cmuresan], Ecosystem QA]

@punam, @mythmon, sorry, it looks like I linked the wrong prefs.js file. The one I should have linked to was only supposed to have changes to the prefs needed to switch to the Normandy Stage server and a keyword.

These are:
user_pref("security.content.signature.root_hash", "DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90");
user_pref("app.normandy.api_url", "https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1");
user_pref("app.normandy.dev_mode", true);
user_pref("app.normandy.logging.level", 0);
user_pref("services.settings.server", "https://settings.stage.mozaws.net/v1");
user_pref("mcoman", true);

I've updated comment 0 with the correct link. The issue we're seeing should be reproducible now. @mythmon, could you please take a look again?

Comment 5

[Michael Cooper [:mythmon]]

The method used in the gif is a standard start up, in which case Normandy specifically waits for the UI to be ready before making any changes. That is expected not to work.

The way to make this work is either by launching from the installer or by passing --first-startup to Firefox on the command line.

The thing that worries me is that according to comment 0, you already tried using the installer to launch Firefox. Can you go into more detail about what you did in that case?

We've also used attribution code and the stub installer method of starting up Firefox for the first time and the issue is reproducible in this way as well. Normandy is either too slow in enrolling or the page is too fast when it is displayed.

Assuming that you were using recipe 931 (public link), this is definitely worrying, since I don't see any thing that should have not worked there.

---

Additionally, it looks like the study did enroll correctly. That makes me wonder if the pref is being read as I understand it, and if the trailhead.firstrun.didSeeAboutWelcome acts as the recipe is expecting. If you set the pref values used in the recipe as default prefs in prefs.js, does that work?

Comment 6

[Ciprian Muresan [:cmuresan], Ecosystem QA]

The method used in the gif is a standard start up, in which case Normandy specifically waits for the UI to be ready before making any changes. That is expected not to work.

I wasn't aware that this was the case, I'll keep it in mind from now on.

The thing that worries me is that according to comment 0, you already tried using the installer to launch Firefox. Can you go into more detail about what you did in that case?

I'd be glad to. I've used two ways of trying to simulate a first run environment, the first one was using a stub installer and the other was using the full installer, in both of these cases I did not manage to be enrolled in the experiment. In both of these cases I've used a clean system (via VM box) where I did not have Firefox installed (initially) and for subsequent attempts I've removed the mozilla folders from AppData (Local and Roaming).
Sadly though, as of today/this evening we can't use stub installers anymore because they install 76.0b1 and the pref is already set to true by default.

If you set the pref values used in the recipe as default prefs in prefs.js, does that work?

This works like a charm and I see the "intended" about:welcome page (at least by the study). Because of this I thought that there was either something wrong with how I approached the scenarios, or it might have been something else.

Let me know if I need to go into greater detail.

Comment 7

[Punam Dahiya [:pdahiya]]

Capturing followup and meeting notes in the bug:

<April 6, 2020> Feedback from mythmon:
Provided build of Firefox that has some extra logging to debug issue: https://treeherder.mozilla.org/#/jobs?repo=try&revision=45ff3f1ae48cb70529ae8fe0cd117854234ae867

Followup today:

• QA able to enroll users and problem not reproduced with above try build.
• mythmon to provide new beta 76 build to simulate enrolling users using installer flow for 'DO No Harm Experiment'

@ciprian Please share your findings. To help debug testing procedure, it will be great if you can provide detail steps followed while testing along with screen recording. Thanks!

Comment 8

[Ciprian Muresan [:cmuresan], Ecosystem QA]

STR I used with both try builds:

1. Cleared my local profiles.
2. Uninstalled any previous installation of Firefox.
3. Installed the try build using the target.installer.exe
4. Did not let the install wizard to automatically open the browser.
5. Navigated to the path I installed Firefox in.
6. Used the following command to open Firefox: .\firefox.exe --first-startup --profile "C:\Users\work\AppData\Roaming\Mozilla\Firefox\Profiles\3tkdnj50.default-beta" --url about:welcome -jsconsole
7. Waited for Firefox to open.

Actual result:

• In the try build that forces the recipe to be run, the simplified about:welcome page is displayed.
• In the try build that DOES NO force the recipe, the default about:welcome page is displayed.

Notes:
Screen recording for the flow with the try build that forces the recipe to be run: link to preview/download
Browser Console output of the same build: link

Screen recording for the flow with the try build that DOES NOT force the recipe: link to preview/download
Browser Console output of the same build: link

The only other thing that comes to mind in this case is the fact that I have used attribution code to filter out the Prod population. I'm not sure if this might have caused this issue to appear, but given that we have seen rather low enrollments in the Chrome Switchers experiment (that used attribution to filter the population) I feel the need to ask. @mythmon could this be a reason why we're seeing this?

Comment 9

[Michael Cooper [:mythmon]]

Based on the information in comment 8, my determination is that this is a problem with attribution filtering, which will be covering in bug 1628409, and that the action of the experiment works fine. We were using attribution filtering for this test because it is an effective way to test the workflow that includes the Firefox installer. It is not needed for the experiment proper.

I think that this bug was caused by a combination of incomplete testing instructions and the bug about attribution filtering. We can go ahead with the experiment, and we'll separately investigate why the attribution filtering isn't working as expected.