1927754 - Ability to ship system-addons out of firefox-main

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement, P3)

Description

[Mike Conley (:mconley) (:⚙️)]

Currently, if we want to update a bundled WebExtension out-of-band via ShipIt / Balrog, we are required to do a little dance where we mirror the extension code over to a github.com/mozilla-extensions repository, and then point ShipIt at that repository.

For extensions that live in github.com/mozilla-extensions, this is fine - but oftentimes, our bundled WebExtensions actually live alongside the Firefox source code in mozilla-central (under browser/extensions). It'd be really lovely if we could point ShipIt at a subfolder of mozilla-central (like browser/extensions/webcompat), and have it package up the XPIs for us, rather than doing the mirroring dance.

I spoke to jlorenzo about this, and apparently, the Releng team agrees! So filing this to get it in the backlog.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Notably, it seems that the xpi-manifest taskcluster job already accepts manifests that include a path to a subdirectory of the repository: https://github.com/mozilla-extensions/xpi-manifest/blob/2375a11832b697d2a51c82d7935d7f5107214b5f/manifests/staging-public.yml#L15-L18

Comment 2

[Mike Conley (:mconley) (:⚙️)]

Hey dschubert - just wanted to get this on your radar: if you look at comment 1, it seems like xpi-manifests can indicate a subdirectory under which their extension is located. I wonder if we could point the webcompat addon repository at mozilla/gecko-dev, and set the subdirectory in the xpi-manifest to browser/extensions/webcompat... would that mean your team wouldn't have to do the github mirroring dance anymore?

Comment 3

[Dennis Schubert [:denschub]]

It would resolve the primary blocker ("ship-it can't even load anything outside the mozilla-extensions GitHub org"), so it would go a long way, but it's not the only thing that needs to happen. Ship-it calls npm run build/yarn run build and expects a .xpi as its artifact. We currently do this with jake, where the environment and job definitions are contained in the GitHub repo, but not in m-c. This can't be a simple call to zip, because after bug 1936031, the addon relies on a bit of preprocessor magic to load a JSON file into a JS file, so this would be a bit more complicated - but absolutely doable.

Comment 4

[Mike Conley (:mconley) (:⚙️)]

Hey jlorenzo,

We spoke about this late last year, and I was wondering if and when there'd be some cycles to get this up and running? We now have the ability to create the webcompat and newtab XPIs from mozilla-central. I think what we need now is some help setting things up in ship-it and xpi-manifests to actually make use of that capability.

Comment 5

[Johan Lorenzo [:jlorenzo]]

Hey :mconley! :ahal started to look into what is needed to make this happen.

Comment 6

[Gabriel Bustamante [:gabriel]]

I think it is possible to point the add-ons pipeline to gecko-dev without too much refactoring.

The UI gets the list of repositories that show up in the UI from xpi-manifest.
I guess we could tweak the entries for the addons here:
https://github.com/mozilla-extensions/xpi-manifest/blob/f2d594ba0ff6544553cddf07456ed548e70aa2b2/taskcluster/config.yml#L81
and make the "default-repository" https://github.com/mozilla/gecko-dev

And run mach as the build system by tweaking xpi manifest here:
https://github.com/mozilla-extensions/xpi-manifest/blob/d39912c79eb900020e593b7541e8fb25060fe3f1/taskcluster/docker/node/build.py#L261-L266

and selecting mach as the install-type on the add-ons xpi manifests:
https://github.com/mozilla-extensions/xpi-manifest/blob/f2d594ba0ff6544553cddf07456ed548e70aa2b2/manifests/webcompat.yml#L9

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: [mozilla-extensions/xpi-manifest] Ability to build out of firefox-main

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: [mozilla-releng/shipit] Support finding xpi versions from firefox-main