1019058 - Incremental build for single app is slow

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Build, defect)

Description

[Alexandre Poirot [:ochameau]]

We don't have datazilla for the build system, but we may easily have regressed build system performances over time. The time it takes to build a simple app feels too long.

For example it takes 9s to build clock app, most of the time is passed in app-makefiles rule of the Makefile (7 to 8s).
It appears that almost half of that time is passed in some Makefile evaluation over here:
https://github.com/mozilla-b2g/gaia/blob/master/Makefile#L522
  $(eval $(call app-makefile-template,$(notdir $(appdir)),$(appdir))) \
Even if we use APP=clock, we eval/call app-makefile-template for all apps and for some reasone, even if we don't execute anything for all of them but clock, it takes several seconds to execute...

By tweaking that particular code, I shrinked down from 9s to 5s clock build time, with only 3s (out of 7/8s) for app-makefiles rules.
I think it would be worth spending some time profiling the build system and coming up with small and surgical patches like that.

Comment 1

[Alexandre Poirot [:ochameau]]

Attachment: Pull request 19882

Comment 2

[Alexandre Poirot [:ochameau]]

Attachment: Pull request 19886

Here is another patch, more complex, but this one comes with more than just performances benefit. It would help revive about:gaia addon by simplifying the Makefile and easing building gaia from just JS script.

Before and ffter executing build script/Makefile per app, we execute a bunch of js script from build/. Before: svoperapps, webapp-manifest, contact-import-services, keyboard-layouts, settings, webapp-shared. After: post-manifest, copy-build-stage-data, webapp-optimize, multilocale, webapp-zip. The key thing here is that there is some small overhead in executing xpcshell many times.

With the previous patch applied and this new one, for APP=clock make, It lower down build time from 4.3s to 3.0s (-30%).

I just asked for feedback as I haven't had time to test/verify this patch.

A natural followup of this patch would be to implement app-makefiles in JS and merge pre-app app-makefiles and post-app into a single one, but that would require getting rid of all app-specific Makefiles. The gain should be even more significant for complete build as we are running one xpcshell instance per app.

Comment 3

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 8432660 [details] [review]
Pull request 19886

I really like this idea for handling dependencies from Makefile to js and that can help about:gaia working again :-)

and that's right we need to get rid of all apps Makefile which is I'm working on bug 1017490.

Comment 4

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 8432619 [details] [review]
Pull request 19882

looks good and low risk for landing in 2.0, r=yurenju

and let's file another follow up for pull request 19886.

Comment 5

[Alexandre Poirot [:ochameau]]

Comment on attachment 8432660 [details] [review]
Pull request 19886

I took some time to test it more and everything looks fine so far.
I just rebased it and resolved a conflict with addition of optimize-clean.

Comment 6

[Alexandre Poirot [:ochameau]]

master:
https://github.com/mozilla-b2g/gaia/commit/e36b232af42b79349a4e0c6c2e1580b437e01e0b