Closed Bug 1462290 Opened 3 years ago Closed 3 years ago

Recipe signature is not valid (for new signed recipes/recipe revisions)

Categories

(Firefox :: Normandy Server, defect, P1)

defect

Tracking

()

VERIFIED FIXED
Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- verified
firefox60 blocking verified
firefox61 blocking verified
firefox62 blocking verified

People

(Reporter: aflorinescu, Assigned: rehan)

References

(Blocks 1 open bug)

Details

(Keywords: qablocker, regression)

Attachments

(1 file)

[Environment:]
Windows 8.1 x64
62.0a1 20180516220130
61.0b5 20180514150347

[Prerequisites:]
1. Set the app.normandy.dev_mode preference to true to run recipes immediately on startup.
2. Set the app.normandy.logging.level preference to 0 to enable more logging.
3. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
4. Set the preference value for app.normandy.api_url set to https://normandy.stage.mozaws.net/api/v1

[Steps:]
I.
   1. Open Control Center (https://normandy-admin.stage.mozaws.net/)
   2. Create a rollout recipe with the blob:
     {
     "preferences": [
     {
     "preferenceName": "test.int.1",
     "value": 1
     }
     ]
     "slug": "rollout-test"
     }
   3. Save, Approve, Publish the recipe.
   4. Set prerequisites and open a Firefox client which supports rollouts (Fx61+)
   5. Open Browser Console and notice the logs.
OR

 II.
   1. Open Control Center (https://normandy-admin.stage.mozaws.net/)
   2. Disable a published recipe.
   3. Publish again the disabled recipe.
   4. Set prerequisites and open a Firefox client which supports rollouts (Fx61+)
   5. Open Browser Console and notice the logs.


[Actual Result:]  
1526549697907	app.normandy.recipe-runner	ERROR	Could not fetch recipes from https://normandy.stage.mozaws.net/api/v1: "Error: recipe signature is not valid"

[Expected Result:]
Recipes should be signed correctly.

[Note:]
1. If all the recipes that have been updated/created today are disabled, Normandy works.
2. The recipe signature error is returned for any type of new/updated recipe (rollout / prefs exp.)
I've just noticed that there has been a Normandy server upgrade, but I'm not sure if this is related to it or a signature issue. Either way, my guess would be that this affects both production and staging environment + all FF clients that run Shield/Normandy.
Summary: Recipe signature is not valid → Recipe signature is not valid (for new signed recipes/recipe revisions)
This may also be related to the Autograph update, which happened recently. CC Ulfr.

I'll look into this. It isn't known if this is affecting prod yet, or why the signatures aren't valid.
This sounds like a blocker.
Severity: major → blocker
Priority: -- → P1
We've determined that this is not related to Autograph, and is likely caused by the recent update to Normandy server. We have a fix incoming.
Assignee: nobody → rdalal
Status: NEW → ASSIGNED
The fix from comment 5 has been merged and deployed to stage. I've verified that the STR from comment 0 no longer cause the problem. The problem still exists on prod, and we'll be deploying the fix there soon.
This has been deployed to prod.
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
I still can reproduce the issue on staging for both new and existing recipes.

For example if you disable/publish recipe https://normandy-admin.stage.mozaws.net/recipe/422/ and run pre-requisites the error is hit again.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Commits pushed to master at https://github.com/mozilla/normandy

https://github.com/mozilla/normandy/commit/79475df13d6796b453e135521c8a4c4860b8c0b3
Bug 1462290 p1 - Adding failing test for signatures during enable

https://github.com/mozilla/normandy/commit/87f2ac282f2fc593e75a1b4b67f5d9aa7d522d45
Bug 1462290 p2 - Refresh recipe during enabling, so it is enabled when signed

https://github.com/mozilla/normandy/commit/a2b146cdb638dd38df9259582a44fcee3d6b42b5
Merge #1382

1382: Bug 1462290 - Correctly update signatures when enabling recipes r=rehandalal a=mythmon

Splitting this into two commits, one with a failing test, and one with the fix.

Co-authored-by: Mike Cooper <mythmon@gmail.com>
Co-authored-by: Rehan Dalal <rehandalal@gmail.com>
Rehan and I found another problem that affects this in a different situation that the one I tested yesterday. We have fixed it, and deployed that fix as v92 to stage.
v92 has now been deployed to prod.

Adrian, can you verify the fix on stage?
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Flags: needinfo?(adrian.florinescu)
(In reply to Michael Cooper [:mythmon] from comment #12)
> v92 has now been deployed to prod.
> 
> Adrian, can you verify the fix on stage?
Sure. Since this issue was server side issue I've verified that the new/update issues are now handled correctly and recipes signed accordingly: Firefox clients (60/61/62) runs new/updated recipes successfully. 

Note that this issue cannot be verified per se on production since we are lacking rights and also it would be bad practice to create test recipes on prod. However, indirectly, there are new (>05.18) functioning recipes on Normandy prod., therefore we can safely assume that the production Normandy deploy fixed the issue there as well.

environment:
60.0.1 20180516032417 (ESR)
61.0b6 20180517141400
62.0a1 20180520220103

Windows 8.1
Ubuntu 16.04
Status: RESOLVED → VERIFIED
Flags: qe-verify+
Flags: needinfo?(adrian.florinescu)
You need to log in before you can comment on or make changes to this bug.