1628409 - Some context data is unneededly not available during first startup runs

Status: RESOLVED FIXED

(Firefox :: Normandy Client, defect, P1)

Description

[Michael Cooper [:mythmon]]

Edit: It seems that this is not about attribution, but other expected context data.

When using the STR from bug 1626904 comment 8, the attribution data does not seem to be available to Normandy filter evaluation during first startup. These STR rely on Normandy recipe 931 being active.

0. [Install a version of Firefox with the target attribution data]
1. Cleared my local profiles.
2. Uninstalled any previous installation of Firefox.
3. Installed the try build using the target.installer.exe [note: this will not overwrite attribution data]
4. Did not let the install wizard to automatically open the browser.
5. Navigated to the path I installed Firefox in.
6. Used the following command to open Firefox: .\firefox.exe --first-startup --profile "C:\Users\work\AppData\Roaming\Mozilla\Firefox\Profiles\3tkdnj50.default-beta" --url about:welcome -jsconsole
7. Waited for Firefox to open.

It appears that the attribution data can also be placed manually, which simplifies the STR significantly.

Comment 1

[Michael Cooper [:mythmon]]

Ciprian, I still can't get this to reproduce locally. In my tests attribution data is always available to Normandy recipes at startup. Can you try and use the same STR that failed before with this Try build?

https://treeherder.mozilla.org/#/jobs?repo=try&revision=22ce73447d21ad4b6848285f59071e493f153fb2

This does not force the execution of 931, and it is a release build, so I don't expect it will work with the recipe anyways. However, it includes some debug code that really breaks down the execution of the filter expression for 931 so that we can see exactly what isn't matching. Importantly, several of the things logged are JS objects. Please make sure to expand out those objects when copying the logs out, especially the filter one marked as "931 filter debug".

Comment 2

[Ciprian Muresan [:cmuresan], Ecosystem QA]

It also dawned on me that I should've linked the stub I used to set the attribution code.

This does not force the execution of 931, and it is a release build, so I don't expect it will work with the recipe anyways.

AFAIK try-builds set the channel name to 'default' and that is one of the filters from 931, so it should work.

I'm not sure how much you wanted me to expand those objects so I've only went with one level down, so that only the first layer of info is visible. Let me know if I should expand this further, the console log is here.

However, I suspect if won't be necessary because from what I can see in the debug log, it's not the attribution code that is not available at initial start-up, it's the 'normandy.os.isWindows' filter. And after the delayed sync time has passed it looks like it gets read accordingly.

Comment 3

[Michael Cooper [:mythmon]]

It turns out that normandy.os has a specific check that it can't work during first startup. Lets fix that.

Comment 4

[Michael Cooper [:mythmon]]

Attachment: Bug 1628409 p2 - Make OS data available in filter context during FirstStartup r=rhelmer

Comment 5

[Michael Cooper [:mythmon]]

Attachment: Bug 1628409 p1 - Move Windows version info determination into a reusable module r=rhelmer

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/19462fc7cca2
p1 - Move Windows version info determination into a reusable module r=rhelmer,chutten,mhowell
https://hg.mozilla.org/integration/autoland/rev/9b655bd7bcb1
p2 - Make OS data available in filter context during FirstStartup r=rhelmer

Comment 7

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/19462fc7cca2
https://hg.mozilla.org/mozilla-central/rev/9b655bd7bcb1

Comment 8

[Ed Lee :Mardak]

It looks like bug 1609070 is also filtering normandy.os.isWindows (in both filter_expression and extra_filter_expression according to https://experimenter.services.mozilla.com/experiments/activity-stream-chrome-switchers-card-for-onboarding-triplets-in-firefox-73-v3/ -> https://normandy.cdn.mozilla.net/api/v3/recipe/927/

We should stop that experiment immediately and we should be able to relaunch it removing any usages of normandy.os.isWindows and replacing with normandy.attribution.ua == "chrome" (as originally used in "v1" https://experimenter.services.mozilla.com/experiments/chrome-switchers-card-for-onboarding-triplets-in-firefox-73/ -> http://normandy.services.mozilla.com/api/v3/recipe/913/ )

mythmon, if we want to relaunch sooner before this fix gets to release, the above should be okay?

Comment 9

[:shell escalante]

(In reply to Ed Lee :Mardak from comment #8)

It looks like bug 1609070 is also filtering normandy.os.isWindows (in both filter_expression and extra_filter_expression according to https://experimenter.services.mozilla.com/experiments/activity-stream-chrome-switchers-card-for-onboarding-triplets-in-firefox-73-v3/ -> https://normandy.cdn.mozilla.net/api/v3/recipe/927/

We should stop that experiment immediately and we should be able to relaunch it removing any usages of normandy.os.isWindows and replacing with normandy.attribution.ua == "chrome" (as originally used in "v1" https://experimenter.services.mozilla.com/experiments/chrome-switchers-card-for-onboarding-triplets-in-firefox-73/ -> http://normandy.services.mozilla.com/api/v3/recipe/913/ )

mythmon, if we want to relaunch sooner before this fix gets to release, the above should be okay?

Ed - the chrome attribute caused issues originally and caused a relaunch https://experimenter.services.mozilla.com/experiments/chrome-switchers-card-for-onboarding-triplets-in-firefox-73-v2/.

I'm guessing you just want the Normandy OS filter removed?

Comment 10

[Su-Young Hong]

I can weigh in here. Removing the OS filter is what's important.

UA filter won't harm the experiment (previous bug with UA targeting originated from the OS bug, not from attribution availability), but it's not necessary either. We can do the following:

• Include UA filter, get about ~25% of the enrollments we would get without UA filter, but they're all chrome attributed
• Not include UA filter, filter out via UA filter in analysis, also get data from people DLing from Edge, etc.

I would recommend the second option.

Comment 11

[Michael Cooper [:mythmon]]

The suggestion looks ok to me, though I heard from Slack that we don't want to relaunch this.

Comment 12

[Ron Manning]

@mythmon, yes we've decided to NOT re-run the Chrome Switchers experiment. The Observation phase was scheduled to end on Apr 28, we are requesting to have it stopped tomorrow.