1462290 - Recipe signature is not valid (for new signed recipes/recipe revisions)

Status: VERIFIED FIXED

(Firefox :: Normandy Server, defect, P1)

Description

[Adrian Florinescu [:aflorinescu]]

[Environment:]
Windows 8.1 x64
62.0a1 20180516220130
61.0b5 20180514150347

[Prerequisites:]
1. Set the app.normandy.dev_mode preference to true to run recipes immediately on startup.
2. Set the app.normandy.logging.level preference to 0 to enable more logging.
3. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
4. Set the preference value for app.normandy.api_url set to https://normandy.stage.mozaws.net/api/v1

[Steps:]
I.
   1. Open Control Center (https://normandy-admin.stage.mozaws.net/)
   2. Create a rollout recipe with the blob:
     {
     "preferences": [
     {
     "preferenceName": "test.int.1",
     "value": 1
     }
     ]
     "slug": "rollout-test"
     }
   3. Save, Approve, Publish the recipe.
   4. Set prerequisites and open a Firefox client which supports rollouts (Fx61+)
   5. Open Browser Console and notice the logs.
OR

 II.
   1. Open Control Center (https://normandy-admin.stage.mozaws.net/)
   2. Disable a published recipe.
   3. Publish again the disabled recipe.
   4. Set prerequisites and open a Firefox client which supports rollouts (Fx61+)
   5. Open Browser Console and notice the logs.


[Actual Result:]  
1526549697907	app.normandy.recipe-runner	ERROR	Could not fetch recipes from https://normandy.stage.mozaws.net/api/v1: "Error: recipe signature is not valid"

[Expected Result:]
Recipes should be signed correctly.

[Note:]
1. If all the recipes that have been updated/created today are disabled, Normandy works.
2. The recipe signature error is returned for any type of new/updated recipe (rollout / prefs exp.)

Comment 1

[Adrian Florinescu [:aflorinescu]]

I've just noticed that there has been a Normandy server upgrade, but I'm not sure if this is related to it or a signature issue. Either way, my guess would be that this affects both production and staging environment + all FF clients that run Shield/Normandy.

Comment 2

[Michael Cooper [:mythmon]]

This may also be related to the Autograph update, which happened recently. CC Ulfr.

I'll look into this. It isn't known if this is affecting prod yet, or why the signatures aren't valid.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

This sounds like a blocker.

Comment 4

[Michael Cooper [:mythmon]]

We've determined that this is not related to Autograph, and is likely caused by the recent update to Normandy server. We have a fix incoming.

Comment 5

[Rehan Dalal (Inactive)]

Fix is here:
https://github.com/mozilla/normandy/pull/1378

Comment 6

[Michael Cooper [:mythmon]]

The fix from comment 5 has been merged and deployed to stage. I've verified that the STR from comment 0 no longer cause the problem. The problem still exists on prod, and we'll be deploying the fix there soon.

Comment 7

[Michael Cooper [:mythmon]]

This has been deployed to prod.

Comment 8

[Adrian Florinescu [:aflorinescu]]

I still can reproduce the issue on staging for both new and existing recipes.

For example if you disable/publish recipe https://normandy-admin.stage.mozaws.net/recipe/422/ and run pre-requisites the error is hit again.

Comment 9

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/normandy/pull/1382

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla/normandy

https://github.com/mozilla/normandy/commit/79475df13d6796b453e135521c8a4c4860b8c0b3
Bug 1462290 p1 - Adding failing test for signatures during enable

https://github.com/mozilla/normandy/commit/87f2ac282f2fc593e75a1b4b67f5d9aa7d522d45
Bug 1462290 p2 - Refresh recipe during enabling, so it is enabled when signed

https://github.com/mozilla/normandy/commit/a2b146cdb638dd38df9259582a44fcee3d6b42b5
Merge #1382

1382: Bug 1462290 - Correctly update signatures when enabling recipes r=rehandalal a=mythmon

Splitting this into two commits, one with a failing test, and one with the fix.

Co-authored-by: Mike Cooper <[e-mail]>
Co-authored-by: Rehan Dalal <[e-mail]>

Comment 11

[Michael Cooper [:mythmon]]

Rehan and I found another problem that affects this in a different situation that the one I tested yesterday. We have fixed it, and deployed that fix as v92 to stage.

Comment 12

[Michael Cooper [:mythmon]]

v92 has now been deployed to prod.

Adrian, can you verify the fix on stage?

Comment 13

[Adrian Florinescu [:aflorinescu]]

(In reply to Michael Cooper [:mythmon] from comment #12)
> v92 has now been deployed to prod.
> 
> Adrian, can you verify the fix on stage?
Sure. Since this issue was server side issue I've verified that the new/update issues are now handled correctly and recipes signed accordingly: Firefox clients (60/61/62) runs new/updated recipes successfully. 

Note that this issue cannot be verified per se on production since we are lacking rights and also it would be bad practice to create test recipes on prod. However, indirectly, there are new (>05.18) functioning recipes on Normandy prod., therefore we can safely assume that the production Normandy deploy fixed the issue there as well.

environment:
60.0.1 20180516032417 (ESR)
61.0b6 20180517141400
62.0a1 20180520220103

Windows 8.1
Ubuntu 16.04