1608832 - Normandy Stage recipes are no longer executed on Beta and Dev Edition 73

Status: VERIFIED FIXED

(Firefox :: Remote Settings Client, defect)

Description

[Patriciu Pop [:ppop], Ecosystem QA]

Attachment: browserconsole.png

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Affected Versions]:

• Firefox Beta 73.0b4, Build ID 20200112220634
• Firefox Dev Edition 73.0b4, Build ID 20200112220634

[Prerequisites]:

• Have the "security.content.signature.root_hash" pref set to "DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90".
• Have the "app.normandy.api_url" pref set to "https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1".
• Have the "app.normandy.dev_mode" pref set to "true".
• Have the "app.normandy.logging.level" pref set to "0".
• Have the "services.settings.server" pref set to "https://settings.stage.mozaws.net/v1".
• Have the "ppop-nor2" pref set to "true".

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites and open the Browser Console.
2. Observe the console output.

[Expected result]:

• The client is enrolled in a recipe.

[Actual result]:

• The client is not enrolled and the "No recipes to execute" string is displayed.

[Notes]:

• Several Remote Settings errors and warnings are displayed in the Browser Console.
• Using the Remote Settings Devtools add-on several timestamps are not synced with the stage server.
• Manually changing the Envrioment to "Stage" causes the dropdown to switch back to "Prod" instantly blocking any attempt to sync to the stage server.
• This issue is not reproducible using Firefox Release 72 and Nightly 74.
• I have attached a screenshot of the browser console:

Comment 1

[Michael Cooper [:mythmon]]

It looks like Remote Settings signatures are failing to validated. I'd expect that to be fine though since RS and Normandy should use the same root hash?

Mat, do you know what's going on here?

Comment 2

[Mathieu Leplatre [:leplatrem]]

Yes indeed, the root hash is the same for the two signing keys.

Are you sure that everything happened sequentially? Couldn't it be that Remote Settings background synchronization happened on the prod server while the root hash was changed?

Is it still happening if you flush the local data completely with the Remote Settings devtools ?

---

Quick side note: Switching Remote Settings to STAGE requires a few additional changes in prefs (for push notifications or load of initial data), and I would recommend using the Dev Tools to do it. (otherwise see the devtools sources)

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 4

[Patriciu Pop [:ppop], Ecosystem QA]

Attachment: RS_devtools_stage.gif

The issue is also reproducible if user.js file with the preferences mentioned in the prerequisites is placed in a newly created profile folder. This would eliminate the synchronization with the prod server as it would point to Stage on start-up.

We cannot switch Remote Settings to Stage using the Dev Tools add-on because the Environment dropdown menu automatically jumps back to "Prod" when selecting "Stage". This is also reproducible on a clean profile without any pref changes, only on Beta and Dev Edition 73.
I have also attached the following .gif highlighting the issue.

Comment 5

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

@Mathieu, could bug 1598562 be the reason why we're having this issue? Or should that bug only affect the release channel? And if so, would that mean that we won't be able to test Normandy recipes on release once that bug gets to 73 Release?

Comment 6

[Mathieu Leplatre [:leplatrem]]

Yes, you're right! This is very likely to be the reason indeed (I don't explain the Dev Tools switching bug though)

Is there a way you could run Firefox with a specific environment variable? At least to test this assumption...

If you run Firefox with this env variable: XPCSHELL_TEST_PROFILE_DIR=/tmp, it should take the server URL from the prefs and read the stage value.

We can then think of something to make your life easier, without removing the hijacking protection of Bug 1598562

Comment 7

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

🎉 Thanks for the suggestion! It worked!

I created a system variable with the XPCSHELL_TEST_PROFILE_DIR name and C:\Users\ciprian.muresan\AppData\Roaming\Mozilla\Firefox\Profiles\ as a value, I assumed that setting it to tmp will point to temporary profiles, and it looks like it reads the pref just fine. I'll still need to figure out how to do this on Mac and Linux, but anything we could do to circumvent this would be much appreciated.

Comment 8

[Mathieu Leplatre [:leplatrem]]

Cool, I'm glad the workaround worked. The Remote Settings code only checks for the presence of this variable, so the value does not matter much.

For Mac & Linux, when you run Firefox from the command-line, you can just put it before the executable: XPCSHELL_TEST_PROFILE_DIR=foo firefox

Are you aware of other parts of Firefox that require a specific flag to do QA tests on Beta/Release?
For security reasons, we don't want to allow the modification of the Remote Settings server via the preferences...

Comment 9

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

Well, the User Journey team have a lot of messages hosted on the Remote Settings server: CFRs, Moments Pages, What's New Panel messages, the Onboarding page has ties to it. This is all I can think of of the top of my head. I assume that if more appear in the future, we can talk about them then.
Would you need me to name the collections that these features host their messages in?

Comment 10

[Mathieu Leplatre [:leplatrem]]

Sorry, my question was not clear :)

I would be happy to mimic other components of Firefox in order to enable some kind of QA or dev mode.

For example, Normandy has the app.normandy.dev_mode pref, but we don't want to rely on prefs.

Maybe there's already a QA_MODE=true environment variable used somewhere else, or something similar.
I wanted to check if you knew other flags for other parts of Firefox, that your teams would already be using for running QA tests. This way we would just use the same ones for Remote Settings.

Comment 11

[Patriciu Pop [:ppop], Ecosystem QA]

@Mathieu Sadly we don't have another component that uses a dev or QA mode in this manner. I've also checked with another team to see if they might know of other flags but we were unsuccessful.

Thank you for your help with this issue! The workaround you provided us was very helpful and we'll surely use it until we have a better alternative.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Just to confirm, there's nothing blocking QA for 73 at this point?

Comment 13

[Mathieu Leplatre [:leplatrem]]

Just to confirm, there's nothing blocking QA for 73 at this point?

I don't think so. I wish the workaround would be more elegant, but from the latest messages I assume they can proceed :)

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:mythmon, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 15

[Patriciu Pop [:ppop], Ecosystem QA]

I've verified that the issue is no longer reproducible when using the workaround provided in Comment 6. Tested using Firefox Beta 74.0b3 and Firefox Release 73.0 on Windows 10 x64, macOS 10.16.2 and Ubuntu 18.