1304782 - sign XML response

Status: RESOLVED MOVED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)

Description

[[e-mail] (:bhearsum)]

This has come up at least a couple of times in the past year. We talked about it a bit around System Addons (IIRC, as a way to help prevent downgrade attacks), and most recently in https://bugzilla.mozilla.org/show_bug.cgi?id=1301956#c20.

Comment 1

[Julien Vehent]

Starting with Firefox 50, the standard way of signing documents pushed to Firefox is to use Content-Signature (CS). CS provides an ecdsa signature using a cert that chains back to the AMO root, trusted in Firefox. The signature must be delivered alongside the signed content, either by HTTP header or embedded in a top-level document.

CC-ing Franziskus who can help with the client code. From the backend side, Balrog Admin will need to call the internal Autograph service [1] to get content signed.

[1] https://mana.mozilla.org/wiki/display/SVCOPS/Autograph+-+Content+Signature

Comment 2

[Daniel Veditz [:dveditz]]

The basic intention behind signing the update XML files is the same as cert pinning the site: The update file is a command (plea?) from our server to the user's browser, and we want to make sure the browser is following our instructions and not those of an interloper. TLS and cert pinning is trying to assure that the user is talking to our server, but it's more important to know that they're getting our content. And pinning in Firefox allows user-installed MITM (practical necessity given the prevelence of network monitoring security devices) so we really don't have much choice about which we should secure.

As it happens we designed the updates to be paranoid about applying the wrong content rather than assuming update.xml is absolutely trustworthy: the downloaded update is signed and we validate the metadata on it. But it would be better to be sure we're fetching the big download from the right source first than to check it after the fact.

It's a defense-in-depth measure in case we've made a mistake in our implementation of the other checks, as it turns out we did with add-on updates.

Comment 3

[Julien Vehent]

We should evaluate the risk of implementing signature on the XML response, and potential reasons for NOT signing it. If the AMO PKI is revoked through a bad OneCRL entry, Firefox clients will reject all content signatures. In this situation, the XML response would be rejected because the content signature would, locking firefox users out of updates.

The same way we don't want to pin the AUS certificates, we may not want to sign the balrog XML response to guarantee we can update users even when things go horribly wrong.

Comment 4

[Julien Vehent]

> If the AMO PKI is revoked through a bad OneCRL entry

This is being addressed in bug 1249002. I think we should move forward with implementing content signature on AUS XML responses.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Julien Vehent [:ulfr] from comment #4)
> > If the AMO PKI is revoked through a bad OneCRL entry
> 
> This is being addressed in bug 1249002. I think we should move forward with
> implementing content signature on AUS XML responses.

It seems a little strange to me that we're talking about using AMO PKI for this. Is that because it happens to be a thing that we have automation built around already, or is there some other reason? From my perspective, it seems like there should be separate keys/certs/infrastructure for AUS response signing.

Comment 6

[Julien Vehent]

It used to be called the AMO PKI because it was the only consumer of it, but nowadays I prefer the term Firefox PKI because it's used for add-ons, kinto collections, and shield recipes.

I would prefer to reuse the current PKI rather than creating a new one, for operational simplicity, but also because the code in firefox would need to be duplicated.

Comment 7

[Robert Strong (they/them - no direct email)]

Will it be possible to only require signing on the xml from aus for system add-ons at first so it is possible to iron out any issues before using it for app updates?

Comment 8

[[e-mail] (:bhearsum)]

A bit more on how this would work in Balrog:
15:00 <bhearsum> how does autograph work? would balrog just feed it the unsigned xml + credentials, and it gets back a signature that goes somewhere in the update response?
15:01 <ulfr> exactly
15:02 <ulfr> then the client needs to verify it
15:02 <ulfr> tricky part is making sure the data you sign and the data that is being verified have the exact same representation
15:02 <ulfr> kinto collections solve that problem with some utf8 sorting of json keys
15:04 <bhearsum> ahhh
15:05 <bhearsum> i don't think that will be a big issue for balrog - mostly we just append strings together to make XML> there's probably a couple of places where we'd have to ensure we're doing this consistently, but not many...

Comment 9

[Julien Vehent]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #7)
> Will it be possible to only require signing on the xml from aus for system
> add-ons at first so it is possible to iron out any issues before using it
> for app updates?

Yes. We can start signing everything in the backend, and only require it for specific types in the client. It might also be useful to make the signature optional via pref in nightly/beta, like we do for add-ons.

Comment 10

[Julien Vehent]

We had a meeting today with myself, :bhearsum, :rstrong, :rhelmer, :catlee, :mostlygeek & :glasserc. The consensus is Content Signature can really help protect the GMP download by signing the hash and URL of the files retrieved from third parties, so we're going to try to move forward with that approach. For now, the only AUS response to be signed is the GMP one, and not the update or system addons since those serve files that are already signed with their own mechanisms.

There is one major concern that needs to be further evaluated: content signatures currently require accurate clocks on clients, to validate the chain of certificates that issued the signature. In the past, we have avoided this requirement when signing updates and add-ons due to clients routinely having bad clocks. Should the signature verification fail, clients wouldn't be able to download media plugins, which has a noticeable impact on their user experience (we currently only use content signature to sign data that doesn't visibly impact user experience, like onecrl). We're going to try to collect more data from clients, possibly via telemetry, to evaluate the size of the population with bad clocks (see also bug 1267229).

Should we estimate that bad clocks are too large a concern, the alternatives are:
1. create a separate signature verification api that ignores certificates validity checks or
2. issue very long lived certificates to sign AUS responses (the firefox pki root is valid from 2015 to 2025, so that's our upper bound).

From the implementation side, Balrog will continue to receive requests directly from Firefox (1), then check its cache for a signed XML document (2) or request autograph to sign an XML document on-the-fly if needed (3). The signed XML is returned to Firefox with a signature in a HTTP Header (4). Firefox then retrieves the chain of certificates from the Content Signature CDN (5), and proceeds to verify the signature, the certificates and their validity (6).


6. verify signature,               2. check local cache
chain of trust & expiration       +--+
        +--+                      |  |
        |  |     1. request AUS   |  |
      +-+--v--+ +-------------> +-+--v-+      +---------+
      | firefox|                 |balrog+----->|autograph|
      +-+--+--+ <-------------+ +------+  3.  +---------+
           |      4. return AUS           request signing of
           |      XML document with       AUX XML document
           |      Content Signature
           |      in HTTP Header
           |
           |             +------------+
           +------------>|Certificates |
           5. retrieve   |CDN         |
           signing certs +------------+

Comment 11

[[e-mail] (:bhearsum)]

Moved to https://github.com/mozilla-releng/balrog/issues/1060