1601901 - [1.128.0] Recipes set up on the Delivery Console Stage are no longer executed on the latest Firefox Release, Beta, or Nightly

Status: VERIFIED FIXED

(Firefox :: Normandy Client, defect, P1)

Description

[Ciprian Muresan [:cmuresan], Ecosystem QA]

Attachment: image.png

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Affected Versions]:

• Firefox Nightly 73.0a1, Build ID 20191205215330
• Firefox Beta 72.0b3, Build ID 20191205201734
• Firefox Release 71.0, Build ID 20191202093317

[Prerequisites]:

• Have the security.content.signature.root_hash pref set to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
• Have the app.normandy.api_url pref set to https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1.
• Have the app.normandy.dev_mode pref set to true.
• Have the app.normandy.logging.level pref set to 0.
• Have the services.settings.server pref set to https://settings.stage.mozaws.net/v1.
• Have a recipe targeting your profile.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Open the Browser Console and observe the output.

[Expected result]:

• The recipe is executed and the user is enrolled in the experiment.

[Actual result]:

• The recipe is not executed and an app.normandy.recipe-runner DEBUG log message is displayed followed by an Argument not a number error.

[Notes]:

• The issue is not reproducible with the prod version of the delivery console.
• The issue was not reproducible yesterday.
• Attached a screenshot of the issue:
• Attached a copy of the Browser Console: Link

@mythmon, @rehan, could any of you please take a look at this?

Comment 1

[Michael Cooper [:mythmon]]

I took a quick look at this, but haven't found a solution yet. I can reproduce it locally on Nightly. The "Argument not a number" error doesn't seem to be coming from any Normandy code, but instead from Telemetry code in C++. I'll see if I can hook up a debugger to this and figure out what's going on. Then I'll try and add better logging for next time this happens.

Comment 2

[Michael Cooper [:mythmon]]

I found the thing that was printing "Argument is not number", and will have a patch up shortly. This is a client bug that makes it so recipes with incompatible capabilities send incorrect telemetry. I'm still not sure why v1.128.0 triggered this, as the recipe that had incompatible capabilities has been active for over a month. More investigation is required. However, I can fix the client problem.

Comment 3

[Michael Cooper [:mythmon]]

Attachment: Bug 1601901 - Check for undefined uptake status in Normandy telemetry r=Chutten!

Comment 4

[Michael Cooper [:mythmon]]

To fill in some details I forgot to migrate from Slack to Bugzilla on Friday: The proximate cause for Argument not a number was a recipe on stage with an unknown-to-the-client capability. That recipe was disabled on Friday, which should have fixed the reported issues.

Comment 5

[Michael Cooper [:mythmon]]

Further investigation with a debugger reveals that there is an unlogged signature error (at least on my system) that is preventing recipe execution.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/76f578f65a4c
Check for undefined uptake status in Normandy telemetry r=chutten

Comment 7

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/76f578f65a4c

Comment 8

[Julien Cristau [:jcristau]]

Should we uplift this to beta?

Comment 9

[Patriciu Pop [:ppop], Ecosystem QA]

I have verified that this issue is no longer reproducible and all Normandy actions are correctly executed in the browser. Tested using Firefox Beta 72.0b4, Firefox Nightly 73.0a1, Firefox Release 71.0 on Windows 10 x64, macOS 10.15.1 and Ubuntu 18.04.

Comment 10

[Michael Cooper [:mythmon]]

The major, blocking problem here was a server-side problem. The "Argument not a number" error was a red herring. It is a good thing to fix, but it doesn't prevent recipes from running. I don't think it needs to be uplifted.