1317768 - Definitive list of what is in OneCRL

Status: RESOLVED WORKSFORME

(Core :: Security Block-lists, Allow-lists, and other State, enhancement, P2)

Description

[Kathleen Wilson]

Please provide a way for everyone to easily see the definitive list of what is in OneCRL.

It would be most excellent if this can be viewed via an https URL that I can copy into my wiki pages, such as https://wiki.mozilla.org/CA:RevokedSubCAcerts

Comment 1

[Rob Stradling]

The OneCRL data is available here (it's an HTTPS URL, but it's not exactly easy to read!:
https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/certificates/records

Since crt.sh already fetches the OneCRL data from the above URL, I figured I may as well create this page (note: it's not quite finished)...

https://crt.sh/mozilla-onecrl

OneCRL doesn't contain the actual cert, but since that's kinda useful information for a "definitive list", this new crt.sh page links to the actual cert (if crt.sh has a copy of it).

Comment 2

[Kathleen Wilson]

Thanks Rob! That is very helpful!

I added both of those links to https://wiki.mozilla.org/CA:RevokedSubCAcerts

I'm going to leave this bug open, because we still need a publicly-readable version-controlled repository that is the source of truth for what we consider to be in OneCRL.

Comment 3

[Rob Stradling]

(In reply to Kathleen Wilson from comment #2)
> Thanks Rob! That is very helpful!
> 
> I added both of those links to https://wiki.mozilla.org/CA:RevokedSubCAcerts

Thanks.  I've completed https://crt.sh/mozilla-onecrl now, but let me know if you'd like to see any further changes to it.

> I'm going to leave this bug open, because we still need a publicly-readable
> version-controlled repository that is the source of truth for what we
> consider to be in OneCRL.

Sure.

Comment 4

[Rob Stradling]

BTW, the 3 certs added to OneCRL in bug #1300747 were added again in bug #1312150.  Please would somebody remove the duplicates?

Comment 5

[Mark Goodwin [:mgoodwin]]

(In reply to Rob Stradling from comment #4)
> BTW, the 3 certs added to OneCRL in bug #1300747 were added again in bug
> #1312150.  Please would somebody remove the duplicates?

Yes, I'll get that sorted.

Comment 6

[Gervase Markham [:gerv]]

(In reply to Rob Stradling from comment #3)
> Thanks.  I've completed https://crt.sh/mozilla-onecrl now, but let me know
> if you'd like to see any further changes to it.

<cough>

> could not find function "x509_name_print" in file "/usr/lib64/postgresql-9.5/lib64/libx509pq.so"
> 
> PL/pgSQL function web_apis(text,text[],text[]) line 1754 at assignment

(As of just now.) But thanks for building this!

Gerv

Comment 7

[Rob Stradling]

(In reply to Gervase Markham [:gerv] from comment #6)
> (In reply to Rob Stradling from comment #3)
> > Thanks.  I've completed https://crt.sh/mozilla-onecrl now, but let me know
> > if you'd like to see any further changes to it.
> 
> <cough>
> 
> > could not find function "x509_name_print" in file "/usr/lib64/postgresql-9.5/lib64/libx509pq.so"
> > 
> > PL/pgSQL function web_apis(text,text[],text[]) line 1754 at assignment

Sorry about that.  I didn't expect crt.sh's front-end servers to be calling that code, because they should be serving a pre-generated version of the page.  I've just tweaked some settings to make it behave properly.

> (As of just now.) But thanks for building this!

You're welcome.  :-)

Comment 8

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

This is an assigned P1 bug without activity in two weeks. 

If you intend to continue working on this bug for the current release/iteration/sprint, remove the 'stale-bug' keyword.

Otherwise we'll reset the priority of the bug back to '--' on Monday, August 28th.

Comment 9

[Gervase Markham [:gerv]]

(In reply to Kathleen Wilson from comment #2) 
> I'm going to leave this bug open, because we still need a publicly-readable
> version-controlled repository that is the source of truth for what we
> consider to be in OneCRL.

Kathleen: can you elaborate on why we need this?

Presumably there's a data store somewhere backing oneCRL. Where is it, and is it publicly available? mgoodwin?

Gerv

Comment 10

[Kathleen Wilson]

(In reply to Gervase Markham [:gerv] from comment #9)
> (In reply to Kathleen Wilson from comment #2) 
> > I'm going to leave this bug open, because we still need a publicly-readable
> > version-controlled repository that is the source of truth for what we
> > consider to be in OneCRL.
> 
> Kathleen: can you elaborate on why we need this?

I think it is good programming practice to have a version-controlled snapshot of each release of OneCRL. And it would be helpful (though not necessary) if it was human-readable. 

Today I use https://crt.sh/mozilla-onecrl and look at the listed Bugzilla bugs to see when an entry was added to OneCRL. This is extremely useful, but it's not really what we should be using as our version-controlled history.


> 
> Presumably there's a data store somewhere backing oneCRL. Where is it, and
> is it publicly available? mgoodwin?


A very small number of us have access to an admin interface where I think we can see the history of when entries were added to OneCRL, but the data is not human-readable, and I think you have to inspect each entry individually. JC has written some scripts that he uses to parse the data in order to verify OneCRL updates before approving them.

Comment 11

[J.C. Jones [:jcj] (he/they)]

This really isn't a PSM bug. It still might necessitate the same set of people, but it should be tracked for blocklists.

Everytime I look at this, I imagine just writing a little script to parse the public Kinto data and present it -- but then I get to comment 10 about checking the version history, and there becomes the problem. So unless said script auto-commits to a Git repository (doable, tho), that would need to scrape the admin interface of Kinto. I'm NI-ing Mattieu to see if he has any ideas on this, but I'm guessing the architecture for getting something out the door for this would be:

script runs periodically
it pulls from https://settings.prod.mozaws.net/v1/buckets/security-state/collections/onecrl/records
it writes out a plaintext file
if any diffs, it commits it to a local git repo
it pushes any commits to Github

That could all be automated pretty tightly, and need no extra permissions.

Comment 12

[Mathieu Leplatre [:leplatrem]]

Note that the official URL is https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/onecrl/records
(the settings.prod.mozaws.net is the origin server and can theorically change)

In order to produce a feed of changes, there is a history endpoint accessible for authenticated users at: https://settings-writer.prod.mozaws.net/v1/buckets/security-state/history?collection_id=onecrl
It could be pulled, filtered and published somewhere public relatively easily I guess.

AFAIU the onecrl collection is populated by a script. This script could also be in charge of publishing something somewhere :)