1328164 - Support |ac_add_options --enable-artifact-builds| / build mode which doesn't require compilation and linking for changes in static files (JS,HTML,XML,XUL)

Status: RESOLVED FIXED

(MailNews Core :: Build Config, enhancement)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Support |ac_add_options --enable-artifact-builds| / build mode which doesn't require compiling and linking for changes in static files (JS,HTML,XML,XUL). Unchanged artifacts should be downloaded from a server.

Bug 901840 added this for Firefox.

Comparison on a Windows machine for a Firefox build after clobber:

without artifact mode: >42 minutes
with artifact mode: <3 minutes

And |mach build faster| after that takes only 6 seconds.

Comment 1

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

See bug 1279781 for a previous attempt to get mach build faster going. Bug 901840 has a lot of dependencies. Do you happen to have a tl;dr version of what needs to be changed in Thunderbird?

Comment 2

[Geoff Lankow (:darktrojan)]

I'm assigning myself this bug as I have something approaching what could be considered "working". I'm going to need to make some changes on mozilla-central and there's also some things on comm-central that break the build.

My current changes needed are here: https://gist.github.com/darktrojan/4e24e58658a682a9d34be408dec7c852

Comment 3

[Geoff Lankow (:darktrojan)]

After many attempts, I have a working Try build. Start-to-finish time: 44 minutes, which would be much less if the decision task didn't take 8 minutes (I think this is because I used a modified mozilla-* tree).

Comment 4

[Geoff Lankow (:darktrojan)]

The next step is to get bug 1444220 fixed.

Comment 5

[Geoff Lankow (:darktrojan)]

Attachment: 1328164-artifact-config-1.diff [landed in comment #11]

This is part one of two. In it I change the mozconfig files so that Try server jobs with --artifact do as expected, and fix up Mac and Windows packaging so that things work correctly in both types of builds.

Comment 6

[Geoff Lankow (:darktrojan)]

Comment on attachment 9072577 [details] [diff] [review]
1328164-artifact-config-1.diff [landed in comment #11]

Scratch that, it's decided to break again. Bah!

Comment 7

[Geoff Lankow (:darktrojan)]

Attachment: 1328164-artifact-config-2.diff

Same again, but also with win32 working. I forgot I hadn't done win32, as linux32 artifact builds just don't work, and mac doesn't have 32-bit any more.

Comment 8

[Geoff Lankow (:darktrojan)]

Comment on attachment 9072584 [details] [diff] [review]
1328164-artifact-config-2.diff

Okay, I give up on win32, it's broken as well. Back to patch one, sorry about the noise.

Comment 9

[Geoff Lankow (:darktrojan)]

Attachment: 1328164-artifact-calendar-1.diff

Part two of two, calendar.

These changes would be mostly unnecessary if it weren't for this problem which makes an artifact build of Lightning unusable. (In a nutshell, the code which compiles a single chrome.manifest for the extension doesn't run in an artifact build without XPI_NAME, but having XPI_NAME and DIST_SUBDIR at the same time produces an even worse result. Instead, we specify XPI_NAME and after the build, copy the files from dist/xpi-stage/lightning to dist/bin/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}.)

This is almost the simplest set of changes I could make that enable both types of build. There's a lot more cleaning up to do but right now I just want to get things moving.

Comment 10

[Rob Lemley [:rjl]]

Comment on attachment 9072577 [details] [diff] [review]
1328164-artifact-config-1.diff [landed in comment #11]

Review of attachment 9072577 [details] [diff] [review]:
-----------------------------------------------------------------

It all looks reasonable enough to me.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/3a3901fff903
Enable artifact builds, part 1: build config; r=rjl

Comment 12

[Geoff Lankow (:darktrojan)]

Attachment: 1328164-artifact-calendar-2.diff

Here's a slightly different version that symlinks Lightning from dist/xpi-stage to dist/bin instead of copying it, which makes development much less annoying.

Comment 13

[Geoff Lankow (:darktrojan)]

Attachment: 1328164-ignore-package-errors-1.diff [landed in comment #15]

I found out something I wish I had known earlier: on mozilla-central, packaging errors are ignored for artifact builds. That's helpful for situations like we currently have with bug 1560545.

Since this bug is still open, I'm reusing it, so this is part three of two.

Comment 14

[Rob Lemley [:rjl]]

Comment on attachment 9074132 [details] [diff] [review]
1328164-ignore-package-errors-1.diff [landed in comment #15]

Review of attachment 9074132 [details] [diff] [review]:
-----------------------------------------------------------------

Neat!

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/f046db4f15a1
Make packaging errors non-fatal for artifact builds. r=rjl

Comment 16

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 9073710 [details] [diff] [review]
1328164-artifact-calendar-2.diff

Review of attachment 9073710 [details] [diff] [review]:
-----------------------------------------------------------------

I'm a bit weary of this patch,  if it were this simple why wouldn't someone have done this before in m-c? I'm ready to try it though.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1afa90fe6710
Enable artifact builds, part 2: calendar packaging. r=philipp

Comment 18

[Geoff Lankow (:darktrojan)]

(In reply to Philipp Kewisch [:Fallen] [:📆] from comment #16)

I'm a bit weary of this patch, if it were this simple why wouldn't someone
have done this before in m-c? I'm ready to try it though.

Which part?

Comment 19

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to Geoff Lankow (:darktrojan) from comment #18)

(In reply to Philipp Kewisch [:Fallen] [:📆] from comment #16)

I'm a bit weary of this patch, if it were this simple why wouldn't someone
have done this before in m-c? I'm ready to try it though.

Which part?

I think it is fine. I was thinking that normally there was a hard split between dist/xpi-stage and dist/bin, but in hindsight I think they are symlinked anyway. I was thinking about that OSX dist thing where they actually copy in to dist/Daily.app/.. so there is a proper application bundle to run (which can't contain symlinks).