Closed Bug 1605733 Opened 5 years ago Closed 5 years ago

Inconsistent records in STAGE after rollback

Categories

(Cloud Services :: Server: Remote Settings, defect, P1)

Product:
Cloud Services
Component:
Server: Remote Settings
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: leplatrem, Assigned: sven)

Details

The collections consistency check fails in STAGE: "blocklists/addons": "status: 'signed'. 7 records present in preview but missing in destination

The 7 records are these ones:

a2e28b6f-e462-3962-d03f-f71e3aec2684
46f75b67-2675-bdde-be93-7ea03475d405
a1554245-da30-8198-c110-bf1f63d485f6
b4e07bae-9d4f-a96b-2ea6-412cfac34375
ffe94023-b4aa-87ac-962c-5beabe34b1a0
2a8b40c7-a1d2-29f4-b7d7-ccfc5066bae1
8514eaee-850c-e27a-a058-8badeeafc26e

See https://delivery-checks.stage.mozaws.net/checks/remotesettings/collections-consistency

According to history:

  • The collections had been signed for weeks, until the 33509a36-d5f1-4dd9-9a54-8cedc0fd3c6d record was created on Thu Dec 12 2019 14:34:30 GMT+0100
  • Review was requested on Thu Dec 12 2019 14:34:34 GMT+0100
  • Rollback was done on Thu Dec 12 2019 16:15:07 GMT+0100

Then, consistency started to fail with the 7 extras records in preview.

When looking at the 7 extra records in preview, we can see that their content is unexpected:

  "data": {
    "deleted": true,
    "id": "a2e28b6f-e462-3962-d03f-f71e3aec2684",
    "last_modified": 1476775382044
  }

This is a bug in the rollback code in kinto-signer :( https://github.com/Kinto/kinto-signer/issues/832

Note that some other bugs in UI have been reported https://github.com/Kinto/kinto-admin/issues/1174

Now that kinto-dist 18.0.2 was deployed (https://bugzilla.mozilla.org/show_bug.cgi?id=1607693), the rollback operation should be fixed and should not lead into this situation.

It would be better to reset this collection completely if possible.

Let's try this, hoping these tombstones records will be nicely deleted:

SERVER=https://settings-writer.stage.mozaws.net/v1

# Patch a random record just to reset the collection status to work-in-progress
echo '{"data":{"enabled":false}}' | http PATCH $SERVER/buckets/staging/collections/addons/records/10425a1e-d517-4c6a-a7f1-330e6599166e -a admin:s3cr3t

# Rollback changes (should reset both preview and staging buckets)
echo '{"data":{"status":"to-rollback"}}' | http PATCH $SERVER/buckets/staging/collections/addons -a admin:s3cr3t

Check that rollback worked properly with this endpoint https://delivery-checks.stage.mozaws.net/checks/remotesettings/collections-consistency
It should be back to sucess:true after at most 2H (because of its TTL)

Thanks!

Assignee: mathieu → sven

I've performed the steps above, and now I'm waiting for the Poucave check to expire.

Maybe we could introduce a query parameter to force running a test? Not particularly important, just a random idea that would be useful in this case.

The Poucave check did not recover – it's still red. Is there anything else I can try? Would you like to get the admin password to the stage instance?

Thanks. That's annoying. The rollback OP didn't do anything. Since those records have this deleted: true field, the rollback code skips them.

So, since it's STAGE and since the bug has been fixed in the deployed versions. Let's fix this with a blowtorch! 🔥🔥

SERVER=https://settings-writer.stage.mozaws.net/v1

http DELETE -a admin:s3cr3t $SERVER/buckets/staging/collections/addons/records/a2e28b6f-e462-3962-d03f-f71e3aec2684
http DELETE -a admin:s3cr3t $SERVER/buckets/staging/collections/addons/records/46f75b67-2675-bdde-be93-7ea03475d405
http DELETE -a admin:s3cr3t $SERVER/buckets/staging/collections/addons/records/a1554245-da30-8198-c110-bf1f63d485f6
http DELETE -a admin:s3cr3t $SERVER/buckets/staging/collections/addons/records/b4e07bae-9d4f-a96b-2ea6-412cfac34375
http DELETE -a admin:s3cr3t $SERVER/buckets/staging/collections/addons/records/ffe94023-b4aa-87ac-962c-5beabe34b1a0
http DELETE -a admin:s3cr3t $SERVER/buckets/staging/collections/addons/records/2a8b40c7-a1d2-29f4-b7d7-ccfc5066bae1
http DELETE -a admin:s3cr3t $SERVER/buckets/staging/collections/addons/records/8514eaee-850c-e27a-a058-8badeeafc26e

Would you like to get the admin password to the stage instance?

Let's try this, and otherwise we can do it together in Brrrlin ;)

Maybe we could introduce a query parameter to force running a test?

I thought of this but discarded it quiickly, mainly because the delivery-checks instances are public, and since some checks are pretty expensive to run, I didn't want to allow this with a simple query param. Now that you say it, using a secret key would work I guess...

Those all result in a 404 for me, e.g.

HTTP/1.1 404 Not Found
Access-Control-Expose-Headers: Alert, Retry-After, Content-Length, Content-Type, Backoff
Connection: keep-alive
Content-Length: 125
Content-Security-Policy: default-src 'none'; frame-ancestors 'none'; base-uri 'none';
Content-Type: application/json
Date: Fri, 17 Jan 2020 15:22:48 GMT
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff

{
    "code": 404,
    "details": {
        "id": "a2e28b6f-e462-3962-d03f-f71e3aec2684",
        "resource_name": "record"
    },
    "errno": 110,
    "error": "Not Found"
}

(I also needed to change the order of the parameters. http does not like to have the auth between method and url).

Oh :( That's my fault... the check output says it all! It's in the preview bucket!

SERVER=https://settings-writer.stage.mozaws.net/v1

http DELETE $SERVER/buckets/blocklists-preview/collections/addons/records/a2e28b6f-e462-3962-d03f-f71e3aec2684 -a admin:s3cr3t
http DELETE $SERVER/buckets/blocklists-preview/collections/addons/records/46f75b67-2675-bdde-be93-7ea03475d405 -a admin:s3cr3t
http DELETE $SERVER/buckets/blocklists-preview/collections/addons/records/a1554245-da30-8198-c110-bf1f63d485f6 -a admin:s3cr3t
http DELETE $SERVER/buckets/blocklists-preview/collections/addons/records/b4e07bae-9d4f-a96b-2ea6-412cfac34375 -a admin:s3cr3t
http DELETE $SERVER/buckets/blocklists-preview/collections/addons/records/ffe94023-b4aa-87ac-962c-5beabe34b1a0 -a admin:s3cr3t
http DELETE $SERVER/buckets/blocklists-preview/collections/addons/records/2a8b40c7-a1d2-29f4-b7d7-ccfc5066bae1 -a admin:s3cr3t
http DELETE $SERVER/buckets/blocklists-preview/collections/addons/records/8514eaee-850c-e27a-a058-8badeeafc26e -a admin:s3cr3t

Sorry Sven :( Maybe next time it'll be more effective if I don't try to give you the commands! 🙄

The updated commands all completed successfully, so now we can wait for the check to update.

Success! https://delivery-checks.stage.mozaws.net/checks/remotesettings/collections-consistency

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.