Closed Bug 1429795 Opened 6 years ago Closed 6 years ago

Please create a new collection in the blocklists bucket in stage / production for certificate preloads

Categories

(Cloud Services :: Operations: Kinto, task)

Product:
Cloud Services
Component:
Operations: Kinto
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mgoodwin, Unassigned)

References

Details

The signer should be as with the current certificate blocklist.

The schema, initially, could look something like this:

{
  "type": "object",
  "description": "A preloaded intermediate certificate.",
  "additionalProperties": false,
  "properties": {
    "blockID": {
      "type": "string",
      "title": "Internal blocklist id",
      "pattern": "^c[0-9]+$",
      "description": "Original block id, eg. c111"
    },
    "details": {
      "type": "object",
      "title": "Details",
      "required": [
        "name",
        "why"
      ],
      "properties": {
        "who": {
          "type": "string",
          "title": "Who"
        },
        "created": {
          "type": "string",
          "format": "date-time"
        },
        "name": {
          "type": "string",
          "title": "Name"
        }
      }
    },
    "id": {
      "type": "string",
      "title": "Id",
      "description": "Record id."
    },
    "schema": {
      "type": "integer",
      "title": "Schema version",
      "description": "Schema version."
    },
    "pubKeyHash": {
      "type": "string",
      "title": "Public key hash",
      "description": "The public key hash of the certificate."
    },
    "subject": {
      "type": "string",
      "title": "Subject name",
      "description": "The certificate subject name."
    },
    "certData": {
      "type": "string",
      "title": "Certificate data",
      "description": "The certificate data."
    },
    "whitelist": {
      "type": "boolean",
      "title": "Whitelist",
      "description": "Should this certificate be whitelisted?"
    }
  },
  "title": "Whitelist Entry",
  "default": {
    "subject": "",
    "pubKeyHash": "",
    "certData": "",
    "whitelist": false,
  },
  "required": [
    "subject",
    "pubKeyHash",
    "whitelist",
  ]
}
Mark, do you think it would make sense to share the ``pinning`` bucket? At least semantically, the preload collection does not seem to be a ``blocklist``.

Should the editors/reviewers group members be the same as pinning/pins?
Flags: needinfo?(mgoodwin)
I think that, semantically, a whitelist isn't much like a list of pins either :)

The editors and reviewers should probably match those for blocklists/certificates
Flags: needinfo?(mgoodwin)
So, following an IRC discussion with :Mathieu, I'd like the following:
1) A new bucket for security state (called something like "security-state" - I'm not fussed about the actual formatting)
2) A new collection in this bucket for intermediate certificate data called "intermediates"

I'd like to re-use the signer we currently use for blocklists/certificates for that collection.

We can migrate the pinning and PKP preload collections to this fx60-ish since we currently have nothing in those collections. Once the work we're doing on CRLite has landed (and our ESR users are on the new mechanism), we can eventually move blocklists/certificates to this bucket too (if we wish).
I have opened a PR to implement these changes. If you have permissions, you can see it at https://github.com/mozilla-services/cloudops-deployment/pull/1520.
Depends on: 1425513
I have problems with data sizes. The set of certificates is approx. 5mb uncompressed. This is okay-ish in itself, but the fact we have to store this *twice* in the profile if we include it in the kinto records is not okay.

I have a solution - but it involves changing the schema so that certData moves to a record attachment. If it's too late for this change (kinto-dist 5.1.4) can we do this in a follow up?
Yes we can still change the schema.
The original PR [1] for this has been merged and deployed to -stage along with kinto-dist v5.1.4.

Are we waiting for more changes to the schema or are we OK to proceed with deploying it to -prod?

Thanks.
We should proceed and we will update the schema once we have it.
The latest permission is now on -prod, along with kinto-dist v5.1.4.
Depends on: 1437692
I have opened https://github.com/mozilla-services/cloudops-deployment/pull/1612 to enable kinto-attachment in the kinto-admin for the security-state-staging/intermediates collection, and disable the certData field.
OK, so for the changes:

I'd like to remove "why" from the required fields (since "why" doesn't make sense in the context of intermediate preloads) and I'd like to move certData (as you mention in comment 11) to an attachment.
Mark, I have updated the PR with the removal of the "why" field, and with exposing additional attachment fields. Because kinto-attachment is already set up for the kinto-settings stack, not much is really "added" here, just some fields which make kinto-admin display an attachment field. Please feel free to review the PR.
:glasserc, PR https://github.com/mozilla-services/cloudops-deployment/pull/1612 is merged, and the change in the permission file has been applied on -stage.

Can some one verify on -stage it's working as expected?

In particular, and iirc, the above PR is supposed to make security-state-staging to share an S3 bucket with 2 others. Can that be verified as well?

Thanks.
Just a note that the permission changes in [1] are deployed to -prod as well. Feel free to test that.

[1] https://github.com/mozilla-services/cloudops-deployment/pull/1612
It turned out that the schema doesn't actually allow attachments on the records. I've opened https://github.com/mozilla-services/cloudops-deployment/pull/1720 to address this. My suspicion is that the files are being uploaded to S3, but then failing to save the record because of the schema.
I've loaded what's in [1] to -stage.

Please test and let us know how it works. If succeeds, we'll then load the same to -prod.

> My suspicion is that the files are being uploaded to S3, ...

Yeah, I can see a new directory called `security-state-staging` has been created in the s3 bucket, on -stage.

[1] https://github.com/mozilla-services/cloudops-deployment/pull/1720
Chatted with Ethan on irc #storage channel. Changes in [1] is merged and applied to -prod.

[1] https://github.com/mozilla-services/cloudops-deployment/pull/1720
Seems to be done on our side. Please reopen if something is missing.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.