1553256 - Kinto Missing OneCRL Entries

Status: RESOLVED INVALID

(Core :: Security: PSM, defect)

Description

[Wayne Thayer]

As described in comment #2 of bug 1548159, some entries found in revocations.txt are not in the legacy certificates or new onecrl bucket in Kinto. An example is most of the entries in bug 1392378. The missing entries nee to be added before the new database storage mechanism in bug 1429796 is released.

Comment 1

[Kathleen Wilson]

Attachment: revocations-May21-2019.txt

Comment 2

[J.C. Jones [:jcj] (he/him)]

I have no idea how we lost entries, presumably during the bucket-cutover? Do we know that's when they were lost, or did they just ... disappear into entropy? Either way that's disturbing.

Comment 3

[Kathleen Wilson]

Note that I have asked Chris Henderson to write script to compare release revocations.txt with
curl https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/onecrl/records | jq '[.data[] | {enabled,issuerName,serialNumber} | select(.enabled) | del(.enabled)]'

So we can get a full list of the deltas.

Comment 4

[Mathieu Leplatre [:leplatrem]]

As long as blocklists/certificates and security-state/onecrl are in sync I can sleep soundly :)
But let me know if we can help here.

Comment 5

[Mathieu Leplatre [:leplatrem]]

You can introspect the records history via HTTP (or via the Admin UI).

For example, the following query will return all creations/modifications of record for a particular issuer:

$ SERVER=https://settings-writer.prod.mozaws.net/v1
$ AUTH="Bearer tehrVFP.....lVC9Uc"
$ http "$SERVER/buckets/staging/history?collection_id=certificates&resource_name=record&target.data.issuerName=MDQxCzAJBgNVBAYTAkZSMRIwEAYDVQQKDAlEaGlteW90aXMxETAPBgNVBAMMCENlcnRpZ25h"  "Authorization: $AUTH"

You can obtain the Bearer token by logging on the Admin UI and looking at network requests in Dev Tools. (we'll improve this in Bug 1552500)

Comment 6

[Julien Cristau [:jcristau] (back April 22)]

Any update on comment 3?

Can we assign someone here to add the missing entries?

Comment 7

[J.C. Jones [:jcj] (he/him)]

Chris, any ETA on the diff?

Comment 8

[Kathleen Wilson]

I'm clearing Chris' neadinfo, because he only consults to Mozilla on Saturday and Sunday mornings, so I don't want him to get spammed by this. Hopefully he will be able to get to it this weekend.

Comment 9

[Kathleen Wilson]

JC, as per the email from Chris, his investigation did not find a difference between the two data sets -- new OneCRL versus previous OneCRL (attached).

The steps to reproduce the potential problem are here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1548159#c2

It is very possible that the curl statement that was originally used to get the list of issuer/serial number revocations for the new OneCRL might have a problem in it, so it might be a good idea to start there (1548159#c2).

It would be fantastic if there are actually no missing entries, and this bug could be closed as invalid. But would be a good idea to be sure.

Also, I don't think this bug needs to be a blocker for Firefox 68. As I understand it, this particular potential problem is not specific to Firefox 68.

Comment 10

[Mathieu Leplatre [:leplatrem]]

Could it be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1559514 ?

Comment 11

[J.C. Jones [:jcj] (he/him)]

It could indeed, Mathieu. If it was a profile that got the prepackaged dataset before having the pref turned on, it's highly likely. That would definitely be Release, possibly Beta on some circumstances. (The imprecision is because this is an interaction between the code running locally, the data in the build, and the data in Kinto. It would take some doing to try and correlate them all.)

Keeping my needinfo.

Comment 12

[J.C. Jones [:jcj] (he/him)]

I agree with Kathleen from Comment 9 that I don't think there's any reason for this to be marked as blocking a particular release, so we should unblock firefox68 from it.

That said, we probably do have mismatches in the field for clients for some periods of time due to Bug 1559514, as updateCertBlocklist is called upon sync, and updates via changes to the services/settings/dumps/security-state/onecrl.json in-package file won't be reflected until there's an update pulled from Kinto. Which should only happen for clients that aren't run for a while, update to a new build, get a new pacakged onecrl.json, and then see they don't need an update from Kinto. More normal usage patterns should generally get the updates via Kinto.

However, notably, brand new installs look like they may not know the OneCRL list, which must be an existing bug that will also be fixed by Bug 1559514.

All-all, I think this one should be marked invalid as we are generally in-sync, except for cases of Bug 1559514, and that bug is where we should discuss actions for the 68 branch, if any.