1546657 - The Normandy stage server recipes are no longer fetched and executed by the browser

Status: VERIFIED FIXED

(Cloud Services :: Operations: Normandy, defect)

Description

[Carmen Fat [:cfat] - Ecosystem QA]

Attachment: normandy.gif

[Affected versions]:

• Firefox Release 66.0.3
• Firefox Beta 67.0b13
• Firefox Nightly 68.0a1

[Affected platforms]:

• Windows 10 x64
• Ubuntu 16.04 x64
• Mac OS X 10.14

[Prerequisites]:

• Set the app.normandy.dev_mode preference to true to run recipes immediately on startup.
• Set the app.normandy.logging.level preference to 0 to enable more logging.
• Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
• Set the preference value for app.normandy.api_url set to https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Open the Browser Console (Ctrl + Shift + J) and observe the references to the recipes.

[Expected result]:

• The Browser Console displays the fetched recipes and their actions.

[Actual result]:

• The Browser Console displays the following error message: "Could not fetch recipes from https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1: "Error: recipe signature is not valid"

[Note]:

• Please see the screen recording attached.

Comment 1

[Brian Pitts]

Hmm. We did successfully force resigned stage against new autograph deploy yesterday. The heartbeat check for recipe and action signatures is passing: https://stage.normandy.nonprod.cloudops.mozgcp.net/__heartbeat__

Comment 2

[Brian Pitts]

This is due to a known issue with Autograph that is being worked on.

As a followup, we should investigate why our health check accepts the signature but the browser does not.

Comment 3

[Michael Cooper [:mythmon]]

Our health checks currently only check that the signature correctly signs the data. They do not check that the signature comes from a trusted source. Firefox is more thorough in its checks. This level of scrutiny had caught many of our previous problems, which were related to signatures and data becoming out of sync.

I can look into improving the heartbeat checks. Brian, can you elaborate on the problems you're having with Autograph?

Comment 4

[Brian Pitts]

This should be working again.

Comment 5

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

Marking as Resolved - Fixed based on comment 4.

Comment 6

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

I have verified that the recipes are now fetched on all platforms.