1693381 - Treat an empty recipe collection as an error

Status: RESOLVED FIXED

(Firefox :: Normandy Client, enhancement, P3)

Description

[Michael Cooper [:mythmon]]

Currently, Normandy retrieves recipes from Remote Settings, and any recipes that are not included in the collection are assumed to be disabled on the server. This is a generally effective strategy, but it has a bad failure case. In certain infrastructure failures, Remote Settings will report an entirely empty collection to Normandy.

At this point, Normandy should assume that an empty recipe collection is an error and refuse to process recipes. This should be treated as a temporary failure, much as signature errors are today.

This has an issue, though. If we eventually turn off Normandy and truly have zero recipes, the clients will not treat that situation correctly. The plan we should use to accommodate this is that in the situation that Normandy is to be shut down, we store a single, "tombstone" record in Normandy's recipe collection. This will remain in the collection for the foreseeable future. It won't have any action, except to serve as a marker that the collection is not empty.

We should additionally have checks (in Poucave) that verifies the collections are not ever empty.

Comment 1

[Michael Cooper [:mythmon]]

Attachment: Bug 1693381 - Treat an empty Normandy recipe collection as an error r=gijs

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ee955a416bce
Treat an empty Normandy recipe collection as an error r=Gijs

Comment 3

[Natalia Csoregi [:nataliaCs]]

Backed out for failures on browser_actions_PreferenceExperimentAction.js and browser_RecipeRunner.js

backout: https://hg.mozilla.org/integration/autoland/rev/834779db451cf13a0bf8608ce0c93ebd1afd683f

push: https://treeherder.mozilla.org/jobs?repo=autoland&revision=ee955a416bcee24cd3fb496239fa7992902d2636&group_state=expanded&selectedTaskRun=IMl45mgOSrm5m2fLjfCo4A.0

failure log:

• https://treeherder.mozilla.org/logviewer?job_id=332948933&repo=autoland&lineNumber=2508

task 2021-03-12T04:04:26.204Z] 04:04:26 INFO - TEST-PASS | toolkit/components/normandy/test/browser/browser_actions_PreferenceExperimentAction.js | The experiment should accumulate a temporary error - [[{"experiment":{"userFacingName":"Test study 37","userFacingDescription":"Test study 37 description","slug":"test-study 37","branch":"control","expired":false,"lastSeen":"2021-03-12T04:04:24.150Z","experimentType":"exp","enrollmentId":"4f7ff5e6-3c83-4763-92a8-0e2b1a4423d5","actionName":"PreferenceExperimentAction","preferences":{"test.study":{"preferenceValue":false,"preferenceType":"boolean","previousPreferenceValue":"undefined","preferenceBranchType":"default"}}},"reason":"no-recipes"}]] deepEqual [[{"experiment":{"userFacingName":"Test study 37","userFacingDescription":"Test study 37 description","slug":"test-study 37","branch":"control","expired":false,"lastSeen":"2021-03-12T04:04:24.150Z","experimentType":"exp","enrollmentId":"4f7ff5e6-3c83-4763-92a8-0e2b1a4423d5","actionName":"PreferenceExperimentAction","preferences":{"test.study":{"preferenceValue":false,"preferenceType":"boolean","previousPreferenceValue":"undefined","preferenceBranchType":"default"}}},"reason":"no-recipes"}]] -
[task 2021-03-12T04:04:26.205Z] 04:04:26 INFO - Leaving test bound testNoRecipes
[task 2021-03-12T04:04:26.206Z] 04:04:26 INFO - Entering test bound testNoRecipes
[task 2021-03-12T04:04:26.206Z] 04:04:26 INFO - Buffered messages finished
[task 2021-03-12T04:04:26.207Z] 04:04:26 INFO - TEST-UNEXPECTED-FAIL | toolkit/components/normandy/test/browser/browser_actions_PreferenceExperimentAction.js | An error should be logged since some recipes were received -
[task 2021-03-12T04:04:26.208Z] 04:04:26 INFO - Stack trace:
[task 2021-03-12T04:04:26.208Z] 04:04:26 INFO - chrome://mochikit/content/browser-test.js:test_ok:1331
[task 2021-03-12T04:04:26.208Z] 04:04:26 INFO - chrome://mochitests/content/browser/toolkit/components/normandy/test/browser/browser_actions_PreferenceExperimentAction.js:testNoRecipes:842
[task 2021-03-12T04:04:26.208Z] 04:04:26 INFO - Console message: [JavaScript Warning: "normandy.recipe_freshness - Truncating float/double number."]
[task 2021-03-12T04:04:26.209Z] 04:04:26 INFO - Console message: [JavaScript Error: "Unknown Collection "main/nimbus-desktop-experiments"" {file: "resource://services-settings/RemoteSettingsClient.jsm" line: 160}]
[task 2021-03-12T04:04:26.209Z] 04:04:26 INFO - UnknownCollectionError@resource://services-settings/RemoteSettingsClient.jsm:160:5
[task 2021-03-12T04:04:26.209Z] 04:04:26 INFO - sync@resource://services-settings/RemoteSettingsClient.jsm:470:13
[task 2021-03-12T04:04:26.209Z] 04:04:26 INFO -
[task 2021-03-12T04:04:26.211Z] 04:04:26 INFO - Console message: [JavaScript Error: "Unknown Collection "main/nimbus-desktop-experiments"" {file: "resource://services-settings/RemoteSettingsClient.jsm" line: 160}]
[task 2021-03-12T04:04:26.211Z] 04:04:26 INFO - UnknownCollectionError@resource://services-settings/RemoteSettingsClient.jsm:160:5
[task 2021-03-12T04:04:26.211Z] 04:04:26 INFO - sync@resource://services-settings/RemoteSettingsClient.jsm:470:13
[task 2021-03-12T04:04:26.211Z] 04:04:26 INFO -
[task 2021-03-12T04:04:26.212Z] 04:04:26 INFO - Leaving test bound testNoRecipes
[task 2021-03-12T04:04:26.213Z] 04:04:26 INFO - GECKO(1562) | MEMORY STAT | vsize 130551163MB | residentFast 2737MB
[task 2021-03-12T04:04:26.213Z] 04:04:26 INFO - TEST-OK | toolkit/components/normandy/test/browser/browser_actions_PreferenceExperimentAction.js | took 1829ms

• https://treeherder.mozilla.org/logviewer?job_id=332949112&repo=autoland&lineNumber=2468

[task 2021-03-12T04:03:29.285Z] 04:03:29 INFO - TEST-PASS | toolkit/components/normandy/test/browser/browser_RecipeRunner.js | run() shouldn't have run yet -
[task 2021-03-12T04:03:29.286Z] 04:03:29 INFO - Console message: UTM:SVC TimerManager:unregisterTimer - id: recipe-client-addon-run
[task 2021-03-12T04:03:29.287Z] 04:03:29 INFO - Console message: UTM:SVC TimerManager:registerTimer - timerID: recipe-client-addon-run interval: 1 skipFirst: false
[task 2021-03-12T04:03:29.287Z] 04:03:29 INFO - Buffered messages finished
[task 2021-03-12T04:03:29.287Z] 04:03:29 INFO - TEST-UNEXPECTED-FAIL | toolkit/components/normandy/test/browser/browser_RecipeRunner.js | Test timed out -
[task 2021-03-12T04:03:29.288Z] 04:03:29 INFO - Console message: 1615521808976 app.normandy.recipe-runner WARN Disabling Shield because app.normandy.api_url is not an HTTPS url: http://example.com.
[task 2021-03-12T04:03:29.288Z] 04:03:29 INFO - Console message: 1615521808982 app.normandy.recipe-runner WARN Disabling Shield because app.normandy.api_url is not set.
[task 2021-03-12T04:03:29.289Z] 04:03:29 INFO - GECKO(1540) | MEMORY STAT | vsize 2813MB | residentFast 289MB | heapAllocated 101MB
[task 2021-03-12T04:03:29.289Z] 04:03:29 INFO - TEST-OK | toolkit/components/normandy/test/browser/browser_RecipeRunner.js | took 45039ms

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fea024be096b
Treat an empty Normandy recipe collection as an error r=Gijs

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/fea024be096b