1959718 - Create a new build project for browser extensions

Status: RESOLVED FIXED

(Firefox Build System :: General, task, P2)

Description

[Andrew Halberstadt [:ahal]]

We are coming up with a plan to build and ship system addons out of mozilla-central. This way teams won't need to sync their system addons over to the mozilla-extensions org in Github.

The details are still being worked out, for instance it's not clear if all system addons will need to live in mozilla-central, or if we'll support both mozilla-central and mozilla-extensions.

The thing that is clear is that we'll need tasks in mozilla-central that:

1. Build the system addon
2. Sign the system addon
3. Push the system addon to archive.mozilla.org
4. Create a release in balrog for the system addon

Basically, copy / share the release pipeline from mozilla-extensions. We'll also want to integrate this with Shipit after the automation is all in place.

Comment 1

[[e-mail] (:bhearsum)]

(taking a best guess at priority/severity - feel free to adjust)

Comment 2

[Andrew Halberstadt [:ahal]]

Hey glandium,

I'm standing up some tasks to build the system addon in CI. Locally I can run, e.g ./mach build browser/extensions/newtab but this runs configure, so the full set of build dependencies are required, where I think in reality we only need the text prepropcessor. Do you have any tips on how to set things up to avoid configure / unnecessary build dependencies in CI?

Comment 3

[Mike Hommey [:glandium]]

This is what I wrote in https://phabricator.services.mozilla.com/D239386?id=999656#inline-1338507:

considering your use case, another option for you would be to create a completely separate "project" rather than build the xpi as a side effect of a full build.

So, like, you'd build with --enable-project=browser/extensions/newtab and after mach build/mach package, you'd only get the xpi out.

So, adding moz.configure and app.mozbuild in browser/extensions/newtab, as well as a few other things.

Comment 4

[Andrew Halberstadt [:ahal]]

Thanks, I'm not sure if this is what it was intended for, but I found building with:
https://searchfox.org/mozilla-central/source/browser/config/mozconfigs/linux64/source

still builds the extensions without the rest of the build. So unless there's a reason not to, I'm planning to use that. (Creating a separate project sounds a bit more complicated than I was hoping)

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1959718 - Create a new build project + variant for browser/extensions, r?#firefox-build-system-reviewers,glandium

Comment 6

[Andrew Halberstadt [:ahal]]

Thanks, I'm not sure if this is what it was intended for, but I found building with:
https://searchfox.org/mozilla-central/source/browser/config/mozconfigs/linux64/source

Oops, this isn't actually true. Using this mozconfig does result in packaged .xpi files, but they are missing a bunch of things (like manifest.json) so are not valid extensions.

Glandium, I made this WIP patch that tries to follow your suggestion:
https://phabricator.services.mozilla.com/D246947

It works, but it still requires the full set of toolchains as part of configure. I'm guessing I need to disable unnecessary pieces explicitly in the moz.configure file? Do you know of any examples, docs or tips that can help me with this?

Comment 7

[Andrew Halberstadt [:ahal]]

Ok, so shortly after I posted this, I happened to stumble across this comment:
https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/mach_commands.py#3406

I tried it out on a whim, and it actually seems to work? E.g, with phab revision above applied, and a mozconfig like:

ac_add_options --enable-project=browser/extensions
ac_add_options --disable-compile-environment
export BUILD_BACKENDS=FasterMake,RecursiveMake

This appears to build the extensions without requiring any toolchains during configure. I guess I could try setting --disable-compile-environment and BUILD_BACKENDS in the moz.configure so a special mozconfig isn't required (beyond --enable-project).

I'm going to leave your needinfo glandium just in case I'm missing something here, after all the comment does warn that you should know what you're doing, and I certainly don't ><.

But I think I have a path forward.

Comment 8

[Mike Hommey [:glandium]]

You should be able to add imply_option("--enable-compile-environment", False) in your browser/extensions/moz.configure. I don't know why you'd need to set BUILD_BACKENDS.

Comment 9

[Nick Alexander :nalexander [he/him]]

(In reply to Mike Hommey [:glandium] from comment #8)

You should be able to add imply_option("--enable-compile-environment", False) in your browser/extensions/moz.configure. I don't know why you'd need to set BUILD_BACKENDS.

It lets you do fancy l10n things, if you know just what you're doing.

Comment 10

[Andrew Halberstadt [:ahal]]

I don't know why you'd need to set BUILD_BACKENDS.

I'll have to re-test to make sure, but without BUILD_BACKENDS the generated .xpi files were missing lots of files. With BUILD_BACKENDS, they were generated correctly. It's possible that something else made the difference and I just thought it was BUILD_BACKENDS.

Comment 11

[Andrew Halberstadt [:ahal]]

I'm going to rename this bug to focus on only the Firefox build system bits, and then use bug 1927754 to track the necessary changes to the release pipeline in xpi-manifest.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/76e46748fabe
Create a new build project + variant for browser/extensions, r=glandium,firefox-build-system-reviewers

Comment 13

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/76e46748fabe