707569 - Speed up |make package|

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mounir Lamouri (:mounir)]

Originally |make package| was only used very rarely to create package for releases so it's speed wasn't a big deal but nowadays it is used to generate android package (.apk) which has to be called every time a developer wants to test a change on his device.
On my laptop doing so requires around 1 minute. This is ridiculously slow compared to the time it takes me to compile a change.

For what I've seen, the main reason of this slowness is that invoking |make package| deletes all outputs files and recreates them. I think we should keep them and only update them if needed.
If our current behavior has some good reasons to exists, we should at least create a fast-package rule that would basically set up a env variable and call |make package| which wouldn't delete everything.

Comment 1

[Mounir Lamouri (:mounir)]

With the patches in the bugs in the dependency list, |make fast-package|, the speed improvement is a bit higher than 50% (goes from around 1m10s to 25-30s).

Comment 2

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

Can we use |make package| instead of |make fast-package|?
The only risk I see is to have some old files you have deleted in your package but it's not really important for release build I guess since they're probably made from scratch.

Comment 3

[Mounir Lamouri (:mounir)]

(In reply to Vivien Nicolas (:vingtetun) from comment #2)
> Can we use |make package| instead of |make fast-package|?
> The only risk I see is to have some old files you have deleted in your
> package but it's not really important for release build I guess since
> they're probably made from scratch.

I'm all for that but I guess some people might not really like the idea so using a different target to begin with looked like a good alternative to me. If khuey says making all changes directly to |make package| is fine, I will do it gladly :)

Comment 4

[Axel Hecht [:Pike]]

When dealing with packaging, be careful to not break being able to run multiple repacks in a row.

I did check the patches, and they seem to only affect the installer-stage target and before, which is fine. The l10n stuff only touches INNER_MAKE_PACKAGE and INNER_UNMAKE_PACKAGE, and after.

Comment 5

[Mike Hommey [:glandium]]

There are only really two things that make most of the time in make package: creating the js shell zip, which is pointless except on tinderbox builds, and xpt.py, which is very slow (bug 654448).

Comment 6

[Justin Wood (:Callek)]

(In reply to Mounir Lamouri (:volkmar) (:mounir) from comment #3)
> (In reply to Vivien Nicolas (:vingtetun) from comment #2)
> > Can we use |make package| instead of |make fast-package|?
> > The only risk I see is to have some old files you have deleted in your
> > package but it's not really important for release build I guess since
> > they're probably made from scratch.
> 
> I'm all for that but I guess some people might not really like the idea so
> using a different target to begin with looked like a good alternative to me.
> If khuey says making all changes directly to |make package| is fine, I will
> do it gladly :)

We'd have problems on dep builds if we did that, fwiw. Especially when a component is moved filenames, and thus the old component name and the new get loaded, name conflicts, test failures, and broken builds will arise. This is not a problem on any clobber build as you said, but MOST devs and almost-all non-nightly builds are non-clobber as well, which is why I feel this is a bad idea...

We could however try and detect a clobber and *ONLY*THAT*BUILD* do make fast-package by default when make package is called... (but future make package calls will always do the slower package) ;-)

Comment 7

[Mike Hommey [:glandium]]

Arguably, there is nothing to optimize for when doing a clobber build...

Comment 8

[Benoit Girard (:BenWa)]

I filed bug 727960 outlining how we can test changes by passing make package.

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per meeting w/ted, joey friday, I'll take this and see what I can do to help.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: have "build_and_deploy" target use new "fast-package"

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 621801 [details] [diff] [review]
have "build_and_deploy" target use new "fast-package"

http://hg.mozilla.org/integration/mozilla-inbound/rev/7d96ff6113c7

Comment 12

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/7d96ff6113c7