1198371 - investigate signing addons used in test harnesses by AMO signing process

Status: RESOLVED DUPLICATE of bug 1283638

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

Currently we run tests that use addons that are unsigned.  When we enable the pref that forces only signed addons to be used, these tests will fail.  We should investigate how we could use the AMO signing process to sign the test addons and include them on an ongoing basis.

Comment 1

[Kim Moir [:kmoir] ET]

Jason, 

:mossop mentioned to me that you are the person to talk to with respect to to the AMO signing process.  We need to investigate how to sign test extensions automatically to be able to run tests on builds where addon signing is enforced.  Do you have any insight on how we could integrate signing them into the AMO signing process?

See bug 1132995 for more details

Comment 2

[Jason Thomas [:jason]]

How many test add-ons are there and do they need to be signed automatically? How often are new test add-ons created?

In the past I have signed test add-ons manually by directly accessing the add-on signing service (trunion app). This is outside of the AMO signing process which requires you to upload the add-on to AMO to be reviewed and then signed https://wiki.mozilla.org/AMO/SigningService#Signing_Architecture. 

I am not sure if AMO has support to sign add-ons in a automated method but I've cc'd a few people that maybe able to answer that.

Any direct access to the add-on signing service (trunion app) outside of AMO would need to be vetted. This would require network ACL to be opened from machines that need to automatically sign add-ons.

Comment 3

[Andrew Halberstadt [:ahal]]

Pasting a quote from jgriffin in an email:

An overview of how our automation uses add-ons:

* some addons are checked into the tree as .xpi files; these are generally trivial ones that don't change much
* other addons are built from source into .xpi files during the build; Ted can probably answer exactly where in the build process this happens
* still other addons are left unpacked, but may be modified by the build process
* the vast majority of test addons are unsigned; a few are signed for test purposes (e.g., see https://dxr.mozilla.org/mozilla-central/search?q=path%3Asigned.xpi&redirect=false&case=false&limit=70&offset=0)
* none of the test addons are installed into the build; they are all installed into the profile by the test harnesses as needed, or loaded from the file system by the tests
* some test addons are changed quite frequently on trunk/aurora, but rarely on beta/release
* there's no difference between desktop and mobile addons as far as the build and the test harnesses are concerned; it's mostly the same set of addons which are used for both
* all the test addon source is inside mozilla-central

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to Jason Thomas [:jason] from comment #2)
> How many test add-ons are there and do they need to be signed automatically?
> How often are new test add-ons created?
> 
There are hundreds of test addons, and they'll need to be signed at each uplift (every 6 weeks), and occasional changes can be expected between those uplifts.  It's too big a task to be left to a manual process, IMO; it will need to be automatable.

Comment 5

[Kim Moir [:kmoir] ET]

Mathieu and Wil, do you have any insight on how we could use the AMO signing process in an automated fashion to sign the addons we use in testing?

Comment 6

[Wil Clouser [:clouserw]]

An API for signing add-ons was determined to be a 2nd priority and dropped from our v1 scope.  It's not on any timelines at this point, the focus now is review queue improvements.

Is it possible to sign with a fake cert for the trunk/aurora add-ons that change so often (but a cert still trusted by trunk/aurora) and only use the real one for release?

Comment 7

[Kim Moir [:kmoir] ET]

That's too bad that there aren't any plans for an API.

One thing to keep in mind in that we are working on release promotion that will allow us to promote CI builds to release channels.  See bug 1118794 for details. So I don't know if separate certs on separate branches is a good way forward, perhaps someone from ateam could chime in on that issue.

So it looks like there will not be an automated way to sign the add-ons in the foreseeable future. Are there any alternatives to this? Are there plans for an API in future quarters?

The original plan when we initially discussed this was to 
run addon tests on the builds on the add-on signing disabled is disabled (add-on-devel builds)
run all other tests on the existing builds with the add-on signing enabled
actually ship existing builds with the add-on signing enabled

Is this the way we should proceed or are there other alternatives?

Comment 8

[Jonathan Griffin (:jgriffin)]

Adding mossop.

Yes, we could use a fake cert on trunk and aurora (as long as the signing was totally integrated with the build process), or not use a cert at all on those branches and keep testing with unsigned addons.

However, for testing on beta and release, we're going to need to build some automation around addon signing.  There are hundreds of test addons, and signing these all manually is going to completely destroy the uplift process, which is complicated enough as-is.  Can we bump the priority of an API for signing, or is there another way we can automate the process?

Comment 9

[Dave Townsend [:mossop]]

Can we just generate a real cert for automation to keep safely and build some process around rather than needing AMO to support an API?

Comment 10

[Kim Moir [:kmoir] ET]

Building a process around signing in our release automation instead using AMO means that we as an organization are maintaining two processes instead of one, and in the long term this consumes more resources.  I echo jgriffin's question regarding is there is a way to bump the priority of an API for signing.

Comment 11

[Kev Needham [:kev]]

So, it sounds like this was a higher priority need than has been mentioned previously. It's been mentioned as something we need to have but can be accommodated by a manual process, but I want to understand if that position has changed.
 
Short term, is batch signing using a manual process a challenge? I am all for automating everything, but I would like to understand "how we can deal with this" while we move to "how we should deal with this".

API for internal use is also a different scope from a signing API for add-on developers as well, and we'd need to sort out workflow and auth processes, as well as understanding how we'll ensure these addons meet Security Policy (or if they do).

Comment 12

[Jonathan Griffin (:jgriffin)]

The answer to this depends in part on what "batch signing using a manual process" looks like; do we already know what that process would be?

Comment 13

[Wil Clouser [:clouserw]]

(In reply to Jonathan Griffin (:jgriffin) from comment #12)
> The answer to this depends in part on what "batch signing using a manual
> process" looks like; do we already know what that process would be?

No.  I'm imagining something like attaching a tarball of add-ons to a bug and someone from ops running a script that untars it, signs everything, and tars it back up, and then reattaching it to the bug.  /off the top of my 8am head :)