1568770 - Hundreds of console errors are displayed after setting up a profile to receive Normandy stage recipes

Status: RESOLVED INVALID

(Firefox :: Remote Settings Client, defect, P5)

Description

[Patriciu Pop [:ppop], Ecosystem QA]

Attachment: Stage_console_errors.gif

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Affected Versions]:

• Firefox Nightly 70.0a1, Build ID 20190724220024

[Prerequisites]:

• Have the "security.content.signature.root_hash" pref set to "DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90".
• Have the "extensions.shield-recipe-client.api_url" pref set to "https://stage.normandy.nonprod.cloudops.mozgcp.net/api/v1".
• Have the "app.normandy.dev_mode" pref set to "true".
• Have the "app.normandy.logging.level" pref set to "0"
• Have the "services.settings.server" pref set to "https://settings.stage.mozaws.net/v1".

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Open the browser console and wait for ~1 minute.
3. Observe the console output.

[Expected result]:

• No errors are displayed.

[Actual result]:

• Several warnings are displayed followed by 400 lines of errors.

[Regression Window]:

• This issue is definitely a regression as it is not reproducible on Beta and Release. I will search for a regression windows as soon as I can.

[Notes]:

• The following console warnings are displayed:
  services.settings: Signature verified failed for main/normandy-recipes. Retry from scratch RemoteSettingsClient.jsm:519
  services.settings: Signature verified failed for main/cfr-fxa. Retry from scratch RemoteSettingsClient.jsm:519
• The following errors are displayed:
  Failed to fetch https://firefox-settings-attachments-cdn.stage.mozaws.net/se…ging/intermediates/50703850-cc99-4156-803a-48fed64fa2df.pem: 403 RemoteSecuritySettings.jsm:604
  Failed to download attachment: undefined RemoteSecuritySettings.jsm:634
• I have attached a screen recording of the issue:

Comment 1

[Patriciu Pop [:ppop], Ecosystem QA]

Michael could you take a look at this one as well?

Comment 2

[Michael Cooper [:mythmon]]

Yesterday from approximately 13:00 PDT to 18:00 PDT there was an outage that caused signing to fail on Remote Settings. We fixed the problem on production, but it is possible that the problem still exists on stage.

Comment 3

[Ethan Glasser-Camp (:glasserc)]

These error messages are not really Normandy-related but have to do with failure to fetch attachments on the security-state/intermediates collection. The URL from the error message is really https://firefox-settings-attachments-cdn.stage.mozaws.net/security-state-staging/intermediates/50703850-cc99-4156-803a-48fed64fa2df.pem. This was computed by:

• taking the attachments base URL from the staging server (https://settings.stage.mozaws.net/v1/) -> https://firefox-settings-attachments-cdn.stage.mozaws.net/
• adding it to the attachment.location from a record -> security-state-staging/intermediates/50703850-cc99-4156-803a-48fed64fa2df.pem

Unfortunately, this record is not from the staging server. It's from prod. You can see by looking at https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/intermediates/records?attachment.location=security-state-staging/intermediates/50703850-cc99-4156-803a-48fed64fa2df.pem vs. https://settings.stage.mozaws.net/v1/buckets/security-state/collections/intermediates/records?attachment.location=security-state-staging/intermediates/50703850-cc99-4156-803a-48fed64fa2df.pem . In other words, the root of the error is that we're trying to get an attachment for a prod record using the staging S3 bucket.

I guess the reason this happened is because we start with data from the prod bucket. Perhaps this comes from a DB dump or maybe we did a sync. Then, we change the server URL to stage. This doesn't clear out the intermediates records, and the records don't track what server they came from, so when it comes time to download attachments, we try to download them from stage. I'm not sure what the best fix is, but for now I'm going to call it a bug in the Remote Settings client.

Comment 4

[Mathieu Leplatre [:leplatrem]]

I guess the reason this happened is because we start with data from the prod bucket. Perhaps this comes from a DB dump or maybe we did a sync. Then, we change the server URL to stage.

Good catch, this seems very likely indeed, but it could also be about JSON dumps.

We recommend switching envs using the Remote Settings Devtools, which make sure that we don't load JSON dumps on STAGE (since they come from prod).

I'm not sure what the best fix is, but for now I'm going to call it a bug in the Remote Settings client.

Switching from PROD to STAGE isn't really a situation that we cover in the Remote Settings client.

Instead I would say that the Remote Settings Dev Tools should clear the local DB when switching envs, and that if users wants to switch envs manually like it's described in this ticket, they should disable JSON dumps loading as well as clearing the local DB if a sync happened before.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:leplatrem, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 7

[Mathieu Leplatre [:leplatrem]]

Should we close this bug as Invalid?

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:leplatrem, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

How much does this interfere with your testing? Unless it makes it actually hard to test I think we'll wontfix this bug.

Comment 10

[Patriciu Pop [:ppop], Ecosystem QA]

Hi Liz, this issue doesn't affect our testing at all, except from creating a lot of noise that we sometimes have to sift through. We logged it as a concern because at that time we didn't know what was causing it. From a QA point of view we would like to see it fixed, but we can manage without it.

Comment 11

[Mathieu Leplatre [:leplatrem]]

Does it happen when using the Remote Settings DevTools?

Comment 12

[Mathieu Leplatre [:leplatrem]]

Closing this for lack of activity/feedback.

Switching from stage to prod (or prod to stage) may require to clear the local data.