1477380 - User branch studies are still set by the early-start pref system, causing study values to be discarded.

Status: VERIFIED FIXED

(Firefox :: Normandy Client, defect, P1)

Description

[Darkspirit]

Win10, Nvidia GTX 1060 3GB

If I create a new profile, create gfx.webrender.all.qualified;true and restart, WebRender is active (about:support).

And this looks like a bug to me:
1.) Create a new profile, set app.normandy.dev_mode to true (or app.normandy.run_interval_seconds to 10), restart.
2.) app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified;true (bold font) and 
gfx.webrender.all.qualified is true (default font weight) have been created. Restart.
3.) It's still Direct3D 11 (Advanced Layers) on about:support. Restart.
4.) Still the same.

Comment 1

[Darkspirit]

I hate my typos. Sorry for spamming.

Win10, Nvidia GTX 1060 3GB

If I create a new profile, create gfx.webrender.all.qualified;true and restart, WebRender is active (about:support).

And this looks like a bug to me:
1.) Create a new profile, set app.normandy.dev_mode to true (or app.normandy.run_interval_seconds to 10), restart.
2.) app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified;true (bold font) and 
gfx.webrender.all.qualified;true (default font weight) have been created. Restart.
3.) It's still Direct3D 11 (Advanced Layers) on about:support. Restart.
4.) Still the same.

Comment 2

[Michael Cooper [:mythmon]]

Thanks for the report. This is an easy thing to miss (the feature being turned on, but not actually active). I'm glad someone caught it so quickly.

The recipe running for WebRender right now is using a default-branch pref study. These are known to be incompatible with features that require a restart. If WebRender needs a restart to be active, then the recipe needs updated to account for that.

This is a limitation of Normandy, but not a bug, so I'm going to move this into the component for Shield studies. Additionally, I'm going to work with the WebRender team to see if we can avoid this in the future.

Comment 3

[Darkspirit]

https://mozilla.logbot.info/gfx/20180720#c15048377
Thank you!

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #1)
> 1.) Create a new profile, set app.normandy.dev_mode to true, restart.

A)
app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified;true
and gfx.webrender.all.qualified;true have been created. Restart. Open about:support.
> Compositing	WebRender
> WEBRENDER	
> opt-in by default: WebRender is an opt-in feature
> available by user: Qualified enabled by pref

or

B)
app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified;false
and gfx.webrender.all.qualified;false have been created. Restart. Open about:support.
> Compositing	Direct3D 11 (Advanced Layers)
> WEBRENDER	
> opt-in by default: WebRender is an opt-in feature

Comment 4

[Darkspirit]

I restarted Nightly some more times, prefs are still true, but WebRender isn't enabled. Strange.

Comment 5

[Darkspirit]

Comment 3 works as described, but after one more restart WebRender doesn't work again.

Comment 6

[Kartikaya Gupta (email:[e-mail])]

For comment 5 - is there some way to verify the behaviour?

Comment 7

[Jeff Muizelaar [:jrmuizel]]

I did a manual telemetry analysis looking at pings with a build id of 20180721100146 and wrQualified = available. I looked at what compositor these pings were using and got: defaultdict(int, {u'basic': 4, u'd3d11': 133, u'webrender': 11}). Which suggests that this is probably not working.

Comment 8

[Michael Cooper [:mythmon]]

When testing this, as well as the value of the preference (expected to be true), it is also useful to know if the preference is set on the user branch or the default branch. In about:config, that can be seen by the boldness of the value. A bold value is on the user-branch, which is required for restartful features like this. Non-bold values are default-branch set, and likely won't work in this case.

Checking just the value isn't directly useful, since on a human timescale, the preference is set immediately. It is only on a code time scale do we see the pref not being set, which is why the boldness of the value (which branch it is on) is useful.

Additionally useful is to check logging. Setting app.normandy.logging.level to 0 before starting a profile is also helpful.

As far as my needinfo: I don't know how to verify that webrender is turned on or off, only if Normandy tried to turn it on. Jan, how did you verify that webrender was on or off?

Comment 9

[Kartikaya Gupta (email:[e-mail])]

The usual mechanism for checking that in-browser is to go to about:support and look at the "Compositing" field in the graphics section. From comment 4 it sounds like the pref doesn't persist? I would assume that when normandy sets a user-branch pref, it would remain set as long as the experiment is active. Can you confirm that?

Comment 10

[Darkspirit]

Attachment: 2018-07-23 17-33-12.mp4

Comment 11

[Darkspirit]

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #10)
> Created attachment 8994257 [details]
> 2018-07-23 17-33-12.mp4

"Target" of Desktop shortcut: "C:\Program Files\Firefox Nightly\firefox.exe" -P wr --no-remote

Comment 12

[Darkspirit]

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #10)
1) Getting normandy user ID + setting dev mode to true
2) Because of dev mode I get the WR study directly after startup: both prefs are true and bold. Have to restart.
3) treatment branch: WR is active: Longer startup (Compiling shaders once per Nightly update). Both prefs still true and bold.
4) treatment branch: no WR. Both prefs are still true, but gfx.webrender.all.qualified is not bold.

Comment 13

[Darkspirit]

and is not listed on about:studies ^^

Comment 14

[Michael Cooper [:mythmon]]

Jan: thanks for the video, that's pretty useful (and I think I learned about an unrelated bug from it). Can you re-do it, including the Normandy logs? They show up in the browser console, if you set `app.normandy.logging.level=0`. This will include data about what Normandy is doing that we simply can't get any other way. If the results are similar enough to the previous video, it may be enough to simply copy/paste the logs to Bugzilla.

It is expected that this type of study will not be listed in about:studies. Preference studies aren't included in the listing there.

Comment 15

[Darkspirit]

(In reply to Michael Cooper [:mythmon] from comment #14)
> Can you re-do it, including the Normandy logs?
Yes, I will rebooot!

> It is expected that this type of study will not be listed in about:studies. Preference studies aren't included in the listing there.
All studies have to be listed on about:studies. That is user expectation. Also, if things suddenly change one could see which study has completed and how to bring it back. Where can I see the following "user facing description"? ;)

(Thomas Elin [:relaas] from bug 1474484 comment 0)
> User facing title of the experiment: WebRender
> User facing description of the experiment: New generation graphics rendering engine

Comment 16

[Darkspirit]

Attachment: steps.txt

(In reply to Michael Cooper [:mythmon] from comment #14)
> If the results are similar enough to the previous video, it may be enough to simply copy/paste the logs to Bugzilla.

Behavior seems to be consistent: WebRender is only enabled in the first Nightly session after "treatment" has created prefs (set to true), and otherwise not.

(comment 12 Nightly session No. 2: I can flip app.normandy.dev_mode back to false there and behavior seems to be the same.)

Comment 17

[Michael Cooper [:mythmon]]

Thanks for the logs. They only showed enrollment activity, and never unenrollments. Given that the preference was being set back to not-user-set (not bold in about:config), that was very confusing. So I went digging.

Normandy's process for *default branch* studies is this: it immediately sets the default branch to the target value, and then it sets a second pref in the `app.normandy.startupExperimentPrefs.` namespace to the same value, but on the user branch. This is because changes to the default branch don't persist between restarts, and all the other storage options delay early startup too much. So then, when Firefox restarts, Normandy reads from the `startupExperimentPrefs` branch as fast as it can, and copies the values over to the default branch. This works well for must studies, but not for things like WebRender.

But that's all default branch studies. It shouldn't affect the study since it was changed to user branch, right? Except it does. It turns out that the system that sets up `startupExperimentPrefs` doesn't care if the study is user branch or default branch. So during startup, the default branch changes to the study value.

This is ok for a moment: since the pref was still true on the user branch while WebRender started, WebRender will still be enabled. The problem comes when Firefox writes the preferences to disk. When Firefox writes its to the profile, it only writes preferences that are different than their defaults. And since Normandy changed the default (in memory) for the preference, the preference is not considered changed, and so its value is not written to disk.

So now on the second restart after the study began, the pref does not have a user set value. It still quickly gets set to the study value on the default branch, but as we know that isn't compatible with WebRender. The effect of all this is that WebRender is on for one run of Firefox, and then turns back off in the next, and stays off. This exactly matches your video.

The fix for this should be simple: when storing the startupExperimentPrefs, we'll skip over any studies that aren't on the default branch. That should fix the above problem.

Comment 18

[Michael Cooper [:mythmon]]

Setting tracking flags. This affects all version of Firefox with pref studies. This doesn't affect ESR52, but only because ESR52 didn't have Normandy Client at all, as far as I can tell. These types of studies are rare, so I don't think this will need uplift to anything.

Comment 19

[Michael Cooper [:mythmon]]

Attachment: Bug 1477380 - Don't set startup prefs for user-branch Normandy pref-experiments r?Gijs

Comment 20

[Michael Cooper [:mythmon]]

It turns out that one of the studies being planned right now, P-OMTP (bug 1476418) was going to run on Beta. We may want to uplift this patch to Beta because of that. Once this patch lands on Nightly, I plan to request Beta uplift.

Comment 21

[Phabricator Automation]

Comment on attachment 8994336 [details]
Bug 1477380 - Don't set startup prefs for user-branch Normandy pref-experiments r?Gijs

:Gijs (he/him) has approved the revision.

https://phabricator.services.mozilla.com/D2305

Comment 22

[:Gijs (he/him)]

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #15)
> (In reply to Michael Cooper [:mythmon] from comment #14)
> > Can you re-do it, including the Normandy logs?
> Yes, I will rebooot!
> 
> > It is expected that this type of study will not be listed in about:studies. Preference studies aren't included in the listing there.
> All studies have to be listed on about:studies. That is user expectation.

I am sympathetic to this point... is there a separate bug on file discussing this? I recently had a confusing experience where the number of default rows in my location bar had changed, and I checked about:studies to see if that was why, and didn't see anything. Now I'm realizing that it might still have been a preference study... but there's no way for me as a user to find out. That's... not great.

Comment 23

[Michael Cooper [:mythmon]]

Regarding about:studies, we already have a bug on file for that, as Jan found: bug 1447499. I agree that it is an issue, and it would be better for things to be there. It also isn't really related to this bug. Let keep any conversation about about:studies in bug 1447499.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/856a8f38f2b0
Don't set startup prefs for user-branch Normandy pref-experiments r=Gijs

Comment 25

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/856a8f38f2b0

Comment 26

[Darkspirit]

I am not able to verify the fix because it looks like the WebRender study has been disabled. Bug 1447499 will be fixed before re-enabling it, right? Thank you! :)

Comment 27

[Michael Cooper [:mythmon]]

We don't currently have any timeline for bug 1447499. However, even if it were fixed, about:studies would not be useful in validating this bug, since about:studies only shows the status of the recipe, not the state of the preferences system or the actual activation of features.

If you'd like to verify this fix now, you can still use the Normandy Devtools [0] to enroll in the study directly, even if it is disabled. (As of now the devtools can't control which branch you get enrolled in, however).

[0]: https://github.com/mozilla/normandy-devtools

Comment 28

[Darkspirit]

(In reply to Michael Cooper [:mythmon] from comment #27)
> If you'd like to verify this fix now, you can still use the Normandy Devtools [0] to enroll in the study directly, even if it is disabled. (As of now the devtools can't control which branch you get enrolled in, however).
> 
> [0]: https://github.com/mozilla/normandy-devtools

Thank you!

Win10, Nvidia GTX 1060
(General observation: There is no longer an app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified pref.)

=== CONTROL GROUP ===
Create a fresh profile. Start it.
Open https://github.com/mozilla/normandy-devtools/releases/download/v0.5.2/normandy-devtools.mozilla.com-0.5.2-signed.xpi, click on "Add" and click on its toolbar icon.
Search for "516 - Pref Flip Study: WebRender V1 Experiment [Bug 1474484]".
Open Browser Console (Ctrl+Shift+J). Click on the Run button of this study.
> 1532543253809	app.normandy.recipe-runner	INFO	Executing recipe "Pref Flip Study: WebRender V1 Experiment [Bug 1474484]" (action=preference-experiment)
> 1532543253809	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.has(prefflip-webrender-1474484)
> 1532543253812	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.start(prefflip-webrender-1474484, control)
> 1532543253812	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.startObserver(prefflip-webrender-1474484)
Close Browser Console.
Open about:config, search for "gfx.webrender.all.qualified". It has been created and is false and bold because we are in the control group.
Restart Nightly. Open about:config, gfx.webrender.all.qualified is still false and bold. Open about:support, "Compositing" is still "Direct3D 11 (Advanced Layers)".
Restart Nightly. Open about:config, gfx.webrender.all.qualified is still false and bold. Open about:support, "Compositing" is still "Direct3D 11 (Advanced Layers)".
Click on the Normandy Devtools icon in the toolbar. Open Browser Console (Ctrl+Shift+J). Click on "Run Normandy".
> [...]
> 1532543532604	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.stop(prefflip-webrender-1474484, {resetValue: true, reason: recipe-not-seen})
> 1532543532604	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.hasObserver(prefflip-webrender-1474484)
> 1532543532604	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.stopObserver(prefflip-webrender-1474484)
Close Browser Console. Open about:config, gfx.webrender.all.qualified is gone.


=== TREATMENT GROUP ===
Create a fresh profile. Start it.
Open https://github.com/mozilla/normandy-devtools/releases/download/v0.5.2/normandy-devtools.mozilla.com-0.5.2-signed.xpi, click on "Add" and click on its toolbar icon.
Search for "516 - Pref Flip Study: WebRender V1 Experiment [Bug 1474484]".
Open Browser Console (Ctrl+Shift+J). Click on the Run button of this study.
(I am again in the control group, so I have to create a new profile and redo steps until I am in the treatment group.)
> 1532543876783	app.normandy.recipe-runner	INFO	Executing recipe "Pref Flip Study: WebRender V1 Experiment [Bug 1474484]" (action=preference-experiment)
> 1532543876784	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.has(prefflip-webrender-1474484)
> 1532543876786	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.start(prefflip-webrender-1474484, treatment)
> 1532543876786	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.startObserver(prefflip-webrender-1474484)
Close Browser Console.
Open about:config, search for "gfx.webrender.all.qualified". It has been created and is true and bold because we are in the treatment group.
Restart Nightly. Open about:config, gfx.webrender.all.qualified is still true and bold. Open about:support, "Compositing" is "WebRender".
And now, to finally verify the fix:
Restart Nightly. Open about:config, gfx.webrender.all.qualified is still true and bold. Open about:support, "Compositing" is "WebRender".
Restart Nightly. Open about:config, gfx.webrender.all.qualified is still true and bold. Open about:support, "Compositing" is "WebRender".
Restart Nightly. Open about:config, gfx.webrender.all.qualified is still true and bold. Open about:support, "Compositing" is "WebRender".
*** This bug is confirmed fixed. ***
Click on the Normandy Devtools icon in the toolbar. Open Browser Console (Ctrl+Shift+J). Click on "Run Normandy".
> [...]
> 1532544134622	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.stop(prefflip-webrender-1474484, {resetValue: true, reason: recipe-not-seen})
> 1532544134622	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.hasObserver(prefflip-webrender-1474484)
> 1532544134622	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.stopObserver(prefflip-webrender-1474484)
Close Browser Console. Open about:config, gfx.webrender.all.qualified is gone.
Restart Nightly. Open about:support, "Compositing" is "Direct3D 11 (Advanced Layers)".


I don't know if relevant, but then I did these steps in the "treatment group" Nightly profile:
Click on the Normandy Devtools icon in the toolbar. Open Browser Console (Ctrl+Shift+J).
Search for "516 - Pref Flip Study: WebRender V1 Experiment [Bug 1474484]".
Click on the Run button of this study.
> 1532545075856	app.normandy.recipe-runner	INFO	Executing recipe "Pref Flip Study: WebRender V1 Experiment [Bug 1474484]" (action=preference-experiment)
> 1532545075856	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.has(prefflip-webrender-1474484)
> 1532545075857	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.get(prefflip-webrender-1474484)
> 1532545075857	app.normandy.normandy-driver.actions	DEBUG	Experiment prefflip-webrender-1474484 has expired, aborting.
This study has expired. I am unable to re-enable this study within this Nightly profile, but I don't need to either, I think?

Comment 29

[Michael Cooper [:mythmon]]

Your steps above look great, thanks for the verification.

> (General observation: There is no longer an app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified pref.)

This is very good, as the existence of that pref is in fact the source of the original problem.

> This study has expired. I am unable to re-enable this study within this Nightly profile, but I don't need to either, I think?

This is expected. Studies can only be enrolled in once per profile by design. It would be nice if the devtools could force re-enroll in a study, but they can't as of now. It is something that may good to add in the future though.

Comment 30

[Michael Cooper [:mythmon]]

Comment on attachment 8994336 [details]
Bug 1477380 - Don't set startup prefs for user-branch Normandy pref-experiments r?Gijs

Approval Request Comment
[Feature/Bug causing the regression]: 1371350
[User impact if declined]: User-branch Normandy studies do not work
[Is this code covered by automated tests?]: Yes
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: It is a small change that has been well verified.
[String changes made/needed]: None

Comment 31

[Jeff Muizelaar [:jrmuizel]]

(In reply to Michael Cooper [:mythmon] from comment #29)
> This is expected. Studies can only be enrolled in once per profile by
> design. It would be nice if the devtools could force re-enroll in a study,
> but they can't as of now. It is something that may good to add in the future
> though.

Does this mean that we need to do a new study for webrender because everyone will have already enrolled but not kept webrender on?

Comment 32

[Liz Henry (:lizzard) (relman/hg->git project)]

I'll wait on mythmon's answer to uplift to beta.

Comment 33

[Jeff Muizelaar [:jrmuizel]]

It seems so. Telemetry of people with buildId 20180725103029 which includes this change has the following breakdown of wrQualified people using WebRender:

{u'basic': 4, u'd3d11': 227, u'webrender': 6})

Comment 34

[Michael Cooper [:mythmon]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #31)
> Does this mean that we need to do a new study for webrender because everyone
> will have already enrolled but not kept webrender on?

A new recipe will need to be made on the server. I believe Rob has already done this. It will affect the timing of the study, but you don't need to create a new PHD or anything like that.

Comment 35

[Darkspirit]

Btw, my main profile on Win10 still has app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified;false (bold) and gfx.webrender.all.qualified;false (not bold). If I "reset" every normandy pref and restart Nightly, both prefs come back in the same appearance.

Comment 36

[Darkspirit]

(In reply to Jan Andre Ikenmeyer [:darkspirit] from comment #35)
They are not saved in prefs.js after I close Nightly. But if I start Nightly they are already there before I get a normandy user ID. 
(The Normandy Devtools addon ran once in this profile before I deleted it yesterday. As I've reset all normandy prefs to their default multiple times I wouldn't expect that it could cause this.)

> 1532634191005	app.normandy.recipe-runner	DEBUG	Enabling Shield
> 1532634192981	app.normandy.recipe-runner	DEBUG	Fetched 56 recipes from the server: Heartbeat: Treatmeat Survey, Accessibility Inspector Tool Dev Edition 62 (Bug 1465195), Heartbeat: Control Survey, Accessibility Inspector Tool Dev Edition 62 (Bug 1465195), Pref Flip Study: WebRender V1.1 Experiment [Bug 1474484], Pref Flip Study: Trusted Recursive Resolver Hotfix v3.0, Nightly [Bug 1475321], Pref Flip Study: Trusted Recursive Resolver Hotfix v2.1, Nightly [Bug 1446404], Pref Flip Study: Spocs Cache Test, Nightly [Bug 1477083], Default Browser Messaging Study, Addon Study: Federated Learning [Bug 1477077], Pref Flip: Savant / Critical Events - Existing Users - US [1457226], Pref Flip: Savant / Critical Events - New Users - DE [1457226], Pref Flip: Savant / Critical Events - New Users - US [1457226], Pref Flip: Savant / Critical Events - Existing Users - DE [1457226], Easy Youtube mp3 Deprecation - DE, Pref Rollout: TLS fallback-limit to 1.3, release 61 [Bug 1473987], Pref Rollout: Retained Display Lists, release 61 [Bug 1467514], Easy YouTube mp3 Deprecation, Youtube Best Downloader, Addon Study: Trusted Recursive Resolver 2, Nightly, ru [Bug 1446404, Addon Study: Trusted Recursive Resolver 2, Nightly, en [Bug1446404, AddonStudy: Trusted Recursive Resolver 2, Nightly, de [Bug 1446404], Addon Study: Trusted Recursive Resolver 2, Nightly, it [Bug1446404], Addon Study: Trusted Recursive Resolver 2, Nightly, fr [Bug 1446404], Hotfix: Disable parallel OMTP for OSX 10.9 - part 3 [Bug 1472308], HB Monitor Survey Quick, Variation 0, HB Monitor Survey Quick, Variation 1, HB Monitor Survey Quick, Variation 2, HB Monitor Survey Quick, Variation 3, HB Monitor Survey Quick, Variation 4, Hotfix: Disable HTTP throttling v2 algorithm [Bug 1462906], Screenshots Survey, Out of Date - ES, Out of Date - PL, Out of Date - FR, Out of Date - RU, Out of Date - DE, Out of Date - PT, Pref Flip: Accessibility Inspector Tool Dev Edition 62 (Bug 1465195), Out of date - EN, Opt-Out Study: Cloud Storage Phase 2 [1450016], Heartbeat: Tab Throbber (20 fps), Beta [Bug 1462158], Heartbeat: Tab Throbber (control), Beta [Bug 1462158], Heartbeat: Tab Throbber (30 fps), Beta [Bug 1462158], Addon Study: Trusted Recursive Resolver, Nightly, en [Bug 1446404], Addon Study: Trusted Recursive Resolver, Nightly, de [Bug 1446404], Addon Study: Trusted Recursive Resolver, Nightly, it [Bug 1446404], Addon Study: Trusted Recursive Resolver, Nightly, fr [Bug 1446404], Addon Study: Trusted Recursive Resolver, Nightly, ru [Bug 1446404], Pref Rollout: TLS fallback-limit to 1.3, beta 61 [Bug 1462164], Pref Flip Study: New Tab Spocs Holdback Test, Release [Bug 1456585], Pref Flip Study: New Tab Spocs Optimization Test,  Release [Bug 1458310], Heartbeat: Send Tabs Preliminary rev1 (release, all en-US), Pref Flip Study: New Bookmark Sync, Nightly 61+ [Bug 1456221], Opt-out - Pioneer Enrollment Study (Bug 1382761), Search Defaults Release 2 - US, Search Defaults Release 2 - CA, Opt-Out Ongoing Satisfaction on Release
> 1532634193274	app.normandy.recipe-runner	DEBUG	Fetched 3 actions from the server: opt-out-study, show-heartbeat, preference-experiment
> 1532634193666	app.normandy.recipe-runner	INFO	Executing recipe "Pref Flip Study: WebRender V1.1 Experiment [Bug 1474484]" (action=preference-experiment)
> 1532634193666	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.has(prefflip-webrender-v1-1-1474484)
> 1532634193666	app.normandy.preference-experiments	DEBUG	PreferenceExperiments.get(prefflip-webrender-v1-1-1474484)
> 1532634193666	app.normandy.normandy-driver.actions	DEBUG	Experiment prefflip-webrender-v1-1-1474484 has expired, aborting.
> 1532634193667	app.normandy.recipe-runner	INFO	Executing recipe "Addon Study: Trusted Recursive Resolver 2, Nightly, en [Bug1446404" (action=opt-out-study)
> 1532634193671	app.normandy.normandy-driver.actions	DEBUG	Enrollment is paused for recipe 484

Comment 37

[Darkspirit]

I noticed that I had another Nightly without a window running in the background. After I've killed everything and deleted prefs.js, those prefs did not came back. Just Nightly things. To be sure, please check if app.normandy.startupExperimentPrefs.gfx.webrender.all.qualified is gone your own wrQualified profile.

Comment 38

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8994336 [details]
Bug 1477380 - Don't set startup prefs for user-branch Normandy pref-experiments r?Gijs

Fix for particular studies, let's uplift to beta.

Comment 39

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/releases/mozilla-beta/rev/340bb17cbcd8

Comment 40

[Adrian Florinescu [:aflorinescu]]

I've reproduced the issue on an affected Nightly version (63.0a1 20180722220044) then proceeded to verify the issue as fixed on:

63.0a1  20180731105217
62.0b13	20180730180407 (not sure why the build id states 07-30) 

Note that the bug will be still reproducible for recipes that have been started on a bad build - (e.g stale user has a june Nightly63 or a 62 b11, the recipe applies before update, then even though it is on a fixed version, given the fact that the recipe was started on the bad build, the recipe will still exibit the issue.
Confirmed with :mythmon that the above note is expected behavior for this fix, hence marking the issue as verified.

Comment 41

[Michael Cooper [:mythmon]]

Comment on attachment 8994336 [details]
Bug 1477380 - Don't set startup prefs for user-branch Normandy pref-experiments r?Gijs

This uplift would enable the study "Default Browser Prompt" [0] to run on 61.0.2.

Approval Request Comment
[Feature/Bug causing the regression]: bug 1371350
[User impact if declined]: User-branch Normandy studies do not work
[Is this code covered by automated tests?]: Yes
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]: Yes. STR above in this bug.
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: It is a small change, well tested and verified.
[String changes made/needed]: None

[0]: https://docs.google.com/document/d/1ERD4IqkKUwtFyelQ9q9IrZvNO4bfhTng1c07oGuWXJQ/edit

Comment 42

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8994336 [details]
Bug 1477380 - Don't set startup prefs for user-branch Normandy pref-experiments r?Gijs

Needed for a forthcoming study planned for 61.0.2. Also, this certainly explains some things with respect to some of our pref-off issues during the cycle! Thanks for adding a test too. Approved for 61.0.2.

Comment 43

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/24dd1a9e2c2c

Comment 44

[Adrian Florinescu [:aflorinescu]]

Verified the fix as well on 61.0.2 candidate 20180807170231 on Windows 8.1, Ubuntu 16.04 and Osx 10.13.