1278974 - Require multiple sign-offs for sensitive changes

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P1)

Description

[Chris AtLee [:catlee]]

Certain types of changes in Balrog should require multiple people to sign off on them. For example pushing a new version live to the release channel.

We should be able to define groups of people so that you need 1 person from group X and 1 person from group B to sign off on a requested action before it takes effect. Adding notifications that an action is waiting for sign-off would be great too.

This can be implemented internally to Balrog initially by recording which authenticated users have requested which actions. Once the Nth person requests a sensitive action, it can take place immediately.

Comment 1

[[e-mail] (:bhearsum)]

From bug 1296635:
(In reply to Ben Hearsum (:bhearsum) from comment #0)
> Currently, anyone with the right permission can make a change to Balrog.
> This means that if someone's account is compromised (particularly someone
> with admin permissions), they can affect updates. We want to mitigate this
> by requiring dual sign off for some types of changes to Balrog. The exact
> scope of what should require dual sign off is undecided, but it could be any
> of:
> - Dual sign off for any change done by a human
> - Dual sign off for any change for certain product+channel combinations (eg:
> Firefox, release)
> - Dual sign off for all changes, period.
> - Something else.
> 
> Implementation is to be determined, but it could tie into some existing or
> other desired work, including:
> - Can/should we implement dual sign off as a special type of Scheduled
> Change.
> - Should we think about implementing a rule change sandbox at the same time
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1141801)
> - Should we think about introducing sets of rule changes at the same time?
> Possibly related to the sandbox.
> 
> I don't want this to spiral out of control into a panacea, but at least
> thinking about how dual sign off may fit into the above is worthwhile to
> avoid boxing ourselves in.

Comment 3

[Richard Barnes [:rbarnes]]

To be honest, I'm less concerned about Balrog, and way more concerned about the signing server.  To some degree, I don't care what goes on Balrog, as long as it's not signed.

Comment 4

[Julien Vehent]

> To be honest, I'm less concerned about Balrog, and way more concerned about the signing server.  To some degree, I don't care what goes on Balrog, as long as it's not signed.

We absolutely need to harden the security of the signing service, but that doesn't remove the need for having multiple layers of security: should the signing service be compromised, access to balrog must remain secure and prevent us from pushing bad updates. Multiple sign-offs is very much needed.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Julien Vehent [:ulfr] from comment #4)
> > To be honest, I'm less concerned about Balrog, and way more concerned about the signing server.  To some degree, I don't care what goes on Balrog, as long as it's not signed.
> 
> We absolutely need to harden the security of the signing service, but that
> doesn't remove the need for having multiple layers of security: should the
> signing service be compromised, access to balrog must remain secure and
> prevent us from pushing bad updates. Multiple sign-offs is very much needed.

To spell this out a bit more, if we have dual sign off for signing but not for Balrog, a bad actor with RelEng (or who has access to a single RelEng account) can push out older versions of Firefox to anyone who hasn't updated yet. If they've also discovered a way to bypass client protection around downgrades, they could even downgrade up-to-date users.

Comment 6

[[e-mail] (:bhearsum)]

I did a lot of planning and design for this the past few weeks. The linked to document (https://docs.google.com/document/d/1Y9_cr2cJ-olldiDHMXhAaWaDMTakcVZP7GrpSX7xEWI/edit#) has my full proposal, which has gone through a lot of review from RelEng, RelMan, and Sec. I've filed all the different parts of the project as dep bugs, although I'm sure at least a few more things will come up as we get to work.

https://bugzilla.mozilla.org/showdependencygraph.cgi?id=1278974&display=tree&rankdir=BT is the best overview of the work I can find. Anything without arrows pointing _to_ it is ready to be worked on.

Comment 7

[[e-mail] (:bhearsum)]

All code required for this is now in production. I'm now working on enabling Required Signoffs as described in https://docs.google.com/document/d/1nWfMW99fea0oV_ZUbdC2uNfmoBZi5TMvsfNgxi0-M0A/edit#heading=h.jdiuq07mtu20

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #7)
> All code required for this is now in production. I'm now working on enabling
> Required Signoffs as described in
> https://docs.google.com/document/d/1nWfMW99fea0oV_ZUbdC2uNfmoBZi5TMvsfNgxi0-
> M0A/edit#heading=h.jdiuq07mtu20

Quick status update: we've enabled Required Signoffs for a test channel, and are waiting on bug 1347212 before we can enable it for any live channels. Once that lands, enabling Required Signoffs is simple and quick.

Comment 9

[[e-mail] (:bhearsum)]

We enabled all the signoffs described in the rollout plan (https://docs.google.com/document/d/1nWfMW99fea0oV_ZUbdC2uNfmoBZi5TMvsfNgxi0-M0A/edit?ts=5971a5f4#) on July 19th. There's at least one non-critical follow-up needed (https://bugzilla.mozilla.org/show_bug.cgi?id=1382665), and we'll track any others that come up in new bugs.