914731 - [Gaia] Parallel builds are broken (again)

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Gabriele Svelto [:gsvelto]]

The optimize-clean rule seem to fail during parallel builds. From a quick glance it seems to me that it needs to happen after webapp-optimize while in the current system it may run in parallel to it leading to the error.

Comment 1

[Gabriele Svelto [:gsvelto]]

Attachment: [PATCH] Serialize the webapp optimization and packaging steps

I'm testing this patch right now, I did a few tens of runs and haven't hit the problem again so I think this should fix the issue.

Long-term it would be better if we could encode these operations (generating optimized l10n files, etc...) as real makefile rules so we have proper dependencies and more parallelism.

Comment 2

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Comment on attachment 802461 [details] [diff] [review]
[PATCH] Serialize the webapp optimization and packaging steps

Yuren, could you review this? Thanks.

Comment 3

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Gabriele, your patch looks good but I would like to know how to reproduce this bug, could you provide the steps?

Comment 4

[Gabriele Svelto [:gsvelto]]

(In reply to Yuren Ju [:yurenju] from comment #3)
> Gabriele, your patch looks good but I would like to know how to reproduce
> this bug, could you provide the steps?

It seems to happen when doing a parallel build with a large number of jobs (and cores/hardware threads to back them up), see bug 810382 comment 7. I could reproduce it on my box but I couldn't find a reliable way to make it happen. In general it seems to happen once every few runs if I touch one of the HTML files in between which causes the corresponding l10n optimized file to be regenerated. But it happens like once every 5-10 runs on my box which has only 2 cores / 4 threads; maybe on a wider box it should happen more often.

Comment 5

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 802461 [details] [diff] [review]
[PATCH] Serialize the webapp optimization and packaging steps

that's not easy to reproduce even I tried -j30 on my box. but this change make dependencies of makefile more clean so I would like to give r+ if this can solve parallel build issue.

Comment 6

[Michael Vines [:m1] [:evilmachines]]

This patch looks good to me too fwiw

Comment 7

[Gabriele Svelto [:gsvelto]]

Thanks for the review!

(In reply to Yuren Ju [:yurenju] from comment #5)
> that's not easy to reproduce even I tried -j30 on my box. but this change
> make dependencies of makefile more clean so I would like to give r+ if this
> can solve parallel build issue.

Yeah, it's hard to reproduce. What I did is to serialize those dependencies so that they happen in the same order as when make is run with -j1.

Comment 8

[Gabriele Svelto [:gsvelto]]

Attachment: [PULL REQUEST] Serialize the webapp optimization and packaging steps

Pointer to Github pull-request

Comment 9

[Gabriele Svelto [:gsvelto]]

Merged in master:

https://github.com/mozilla-b2g/gaia/commit/bc795d7e3d10c8295c1368704a46926dcd0ca1de

Fingers crossed in the hope we don't hit more of these issues in Gaia.

Comment 10

[Gabriele Svelto [:gsvelto]]

(In reply to Michael Vines [:m1] [:evilmachines] from comment #6)
> This patch looks good to me too fwiw

If you hit the problem again please re-open this bug. On my side I've being building Gaia since yesterday with this patch applied and I haven't hit the issue anymore.