Closed Bug 1601901 Opened 3 years ago Closed 3 years ago

[1.128.0] Recipes set up on the Delivery Console Stage are no longer executed on the latest Firefox Release, Beta, or Nightly

Categories

(Firefox :: Normandy Client, defect, P1)

Desktop
All
defect

Tracking

()

VERIFIED FIXED
Firefox 73
Tracking Status
firefox71 --- wontfix
firefox72 --- wontfix
firefox73 --- verified

People

(Reporter: cmuresan, Assigned: mythmon)

References

Details

Attachments

(2 files)

Attached image image.png

[Affected Platforms]:

  • All Windows
  • All Mac
  • All Linux

[Affected Versions]:

  • Firefox Nightly 73.0a1, Build ID 20191205215330
  • Firefox Beta 72.0b3, Build ID 20191205201734
  • Firefox Release 71.0, Build ID 20191202093317

[Prerequisites]:

  • Have the security.content.signature.root_hash pref set to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
  • Have the app.normandy.api_url pref set to https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1.
  • Have the app.normandy.dev_mode pref set to true.
  • Have the app.normandy.logging.level pref set to 0.
  • Have the services.settings.server pref set to https://settings.stage.mozaws.net/v1.
  • Have a recipe targeting your profile.

[Steps to reproduce]:

  1. Open the browser with the profile from prerequisites.
  2. Open the Browser Console and observe the output.

[Expected result]:

  • The recipe is executed and the user is enrolled in the experiment.

[Actual result]:

  • The recipe is not executed and an app.normandy.recipe-runner DEBUG log message is displayed followed by an Argument not a number error.

[Notes]:

  • The issue is not reproducible with the prod version of the delivery console.
  • The issue was not reproducible yesterday.
  • Attached a screenshot of the issue:
  • Attached a copy of the Browser Console: Link

@mythmon, @rehan, could any of you please take a look at this?

Flags: needinfo?(rdalal)
Flags: needinfo?(mcooper)

I took a quick look at this, but haven't found a solution yet. I can reproduce it locally on Nightly. The "Argument not a number" error doesn't seem to be coming from any Normandy code, but instead from Telemetry code in C++. I'll see if I can hook up a debugger to this and figure out what's going on. Then I'll try and add better logging for next time this happens.

Depends on: 1602036
Blocks: 1595604

I found the thing that was printing "Argument is not number", and will have a patch up shortly. This is a client bug that makes it so recipes with incompatible capabilities send incorrect telemetry. I'm still not sure why v1.128.0 triggered this, as the recipe that had incompatible capabilities has been active for over a month. More investigation is required. However, I can fix the client problem.

Flags: needinfo?(rdalal)
Flags: needinfo?(mcooper)
Assignee: nobody → mcooper
Status: NEW → ASSIGNED
Priority: -- → P1

To fill in some details I forgot to migrate from Slack to Bugzilla on Friday: The proximate cause for Argument not a number was a recipe on stage with an unknown-to-the-client capability. That recipe was disabled on Friday, which should have fixed the reported issues.

Further investigation with a debugger reveals that there is an unlogged signature error (at least on my system) that is preventing recipe execution.

See Also: → 1602560
Pushed by mcooper@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/76f578f65a4c
Check for undefined uptake status in Normandy telemetry r=chutten
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 73

Should we uplift this to beta?

Flags: needinfo?(mcooper)

I have verified that this issue is no longer reproducible and all Normandy actions are correctly executed in the browser. Tested using Firefox Beta 72.0b4, Firefox Nightly 73.0a1, Firefox Release 71.0 on Windows 10 x64, macOS 10.15.1 and Ubuntu 18.04.

The major, blocking problem here was a server-side problem. The "Argument not a number" error was a red herring. It is a good thing to fix, but it doesn't prevent recipes from running. I don't think it needs to be uplifted.

Flags: needinfo?(mcooper)
You need to log in before you can comment on or make changes to this bug.