Closed Bug 1132995 Opened 5 years ago Closed 3 years ago

Automated testing harness extensions need to be signed to be able to run in Beta and Release builds

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mossop, Unassigned)

References

Details

(Whiteboard: [releng])

When the add-on signing requirement is turned on all our test harnesses which currently install as extensions will need to be signed in order to work in release and beta builds.
Dave, I assume that this not only applies to the harness itself but also to any XPI files as used by tests? That would be a lot of work I assume.
Flags: needinfo?(dtownsend)
(In reply to Henrik Skupin (:whimboo) from comment #1)
> Dave, I assume that this not only applies to the harness itself but also to
> any XPI files as used by tests? That would be a lot of work I assume.

That's right. I've just filed bug 1133838 to track the browser-chrome and xpcshell tests in toolkit/mozapps/extensions specifically but I guess there are others elsewhere and the solution may be different for each.
Flags: needinfo?(dtownsend)
Flags: firefox-backlog+
This is likely primarily to be a releng task.
Component: General → General Automation
Flags: firefox-backlog+
Product: Testing → Release Engineering
QA Contact: catlee
See Also: → 1135781
Blocks: 1149656
No longer blocks: signed-addons
Looks like basically all the test harnesses fail when signing is enabled: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c56931f79aea
Blocks: 1176641
No longer blocks: 1176641
Kim, is this something you're working on?
Flags: needinfo?(kmoir)
One thing we have to be careful of here is that the signed harness extensions aren't something that someone can take and use to inject code into Firefox
(In reply to Dave Townsend [:mossop] from comment #6)
> One thing we have to be careful of here is that the signed harness
> extensions aren't something that someone can take and use to inject code
> into Firefox

Do you have suggestions on how to accomplish that?
The plan is to have signed extensions uplifted as part of the marge process but I have not been actively working on this, finishing up work on bug 1135781 first, then will move on to tests.
Flags: needinfo?(kmoir)
(In reply to Jonathan Griffin (:jgriffin) from comment #7)
> (In reply to Dave Townsend [:mossop] from comment #6)
> > One thing we have to be careful of here is that the signed harness
> > extensions aren't something that someone can take and use to inject code
> > into Firefox
> 
> Do you have suggestions on how to accomplish that?

If the tests are being included in the signed extension then that is probably enough. I'd be concerned if we're signing the test harness and then it was loading code to run from outside the signed extension.
Tests are generally not included in extensions atm; packaging the harnesses as extensions complete with tests would be a significant change from current procedure, and would make updating tests on beta/release very difficult, unless the test extension signing was part of the normal build process.

This isn't likely achievable by the next uplift in Sept; what we should probably target is having the tests run at all using signed addons, and then look at hardening our approach for subsequent releases.

I'll set up a meeting next week so we can discuss this.
(In reply to Jonathan Griffin (:jgriffin) from comment #10)
> Tests are generally not included in extensions atm; packaging the harnesses
> as extensions complete with tests would be a significant change from current
> procedure, and would make updating tests on beta/release very difficult,
> unless the test extension signing was part of the normal build process.
> 
> This isn't likely achievable by the next uplift in Sept; what we should
> probably target is having the tests run at all using signed addons, and then
> look at hardening our approach for subsequent releases.

Ok. A couple of things we could do to mitigate this would be to make sure we give the harness extension a specific version for each Firefox version, that way we can blocklist older versions if we become concerned. Also marking the min/maxVersions for the extension to match the Firefox version they are built for and include the strictCompatibility flag so they can't be used on other versions.
Depends on: 1198371
We had a meeting yesterday on this issue, here are the notes
https://etherpad.mozilla.org/addon-signing-continous-integration-strategy

is summary Kev to talk with mossop re enabling dev mode in Firefox which would allow us to test the addons without the need for implementing another build type or massive refactoring of test harnesses
Depends on: 1219442
Depends on: 1219445
Depends on: 1219446
Depends on: 1220593
Whiteboard: [releng]
Blocks: 1233200
Can this be closed now?
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.