1799521 - Test that codesign validates Firefox installs and updates

Status: NEW

(Toolkit :: Application Update, task, P3)

Description

[Robin Steuber (she/her) [:bytesized]]

We should test that Firefox is properly verified with codesign on macOS, both as a regular installation, and when updated.

There are some unanswered questions on how exactly we will do this

• Presumably Firefox will need to actually be signed for this to work. I'm not sure how to specify that a test should only be run if Firefox is signed properly.
• What version do we update from and how do we get it? We might be able to just take an arbitrary static install, generate a complete MAR, and just call the updater binary. But this wouldn't be a perfect simulation, in part because it would ignore the concept of watershed updates.

The purpose of this is to avoid issues like Bug 1431342. Especially since macOS is apparently now being more rigid about verifying signatures (See Bug 1799332). But doing this means we should probably consider the question of "what do we update from" carefully. When we have hit this issue in the past, the problem did not necessarily occur regardless of what version we update from.

Comment 1

[[e-mail] (:bhearsum)]

This might be something better suited to be done as part of nightly and/or release automation than a regular test. We already have the concept of "update verify" tests that ensure the installer and the updater get you to the same place. These days, those all run on Linux, but it's certainly possible in theory to have similar tests that run on macOS machines.

One complication no matter how we do this is going to be that it's impractical to run any sort of test in CI or automation that's done on a very very up-to-date machine. It usually takes one to two quarters at minimum to upgrade to a new macOS version. So, even if we had these tests, I don't know that we would catch the bleeding edge stuff (we could certainly prevent regressing more longstanding things though).

Comment 2

[Robin Steuber (she/her) [:bytesized]]

(In reply to [e-mail] (:bhearsum) from comment #1)

One complication no matter how we do this is going to be that it's impractical to run any sort of test in CI or automation that's done on a very very up-to-date machine.

I don't think that should be too big of an issue. I believe that codesign will work regardless. The problem here is that codesign not verifying successfully has more serious consequences on newer versions. But I don't think it is doing the signature verification any differently.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #2)

(In reply to [e-mail] (:bhearsum) from comment #1)

One complication no matter how we do this is going to be that it's impractical to run any sort of test in CI or automation that's done on a very very up-to-date machine.

I don't think that should be too big of an issue. I believe that codesign will work regardless. The problem here is that codesign not verifying successfully has more serious consequences on newer versions. But I don't think it is doing the signature verification any differently.

If older codesign tests for things that are only enforced on later versions -- yeah. Surely at some point a new check will be invented that needs a newer codesign to verify though? (Maybe this will still catch 99% of issue though...which would be great!)