1261283 - When compiling with high -j count, compilation will often fail

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Jean-Yves Avenard [:jya]]

I was experimenting with distcc

When setting .mozconfig with:
mk_add_options MOZ_MAKE_FLAGS="-s -j48"
CC="distcc clang -target x86_64-apple-darwin"
CXX="distcc clang++ -target x86_64-apple-darwin"

I often get the compilation to fail with:

 0:40.10 gmake[5]: *** No rule to make target 'xpidlyacc.py', needed by 'export'.  Stop.
 0:40.10 /Users/jyavenard/Work/Mozilla/mozilla-central/config/recurse.mk:79: recipe for target 'xpcom/idl-parser/xpidl/export' failed
 0:40.10 gmake[4]: *** [xpcom/idl-parser/xpidl/export] Error 2
 0:40.10 gmake[4]: *** Waiting for unfinished jobs....

Comment 1

[Mike Hommey [:glandium]]

This happens on automation as well.

Comment 2

[Michael Shal [:mshal]]

I think we can fix this by adding support for GENERATED_FILES with multiple outputs - I'm testing out a patch now. Or we could temporarily add the xpidlyacc.py: xpidllex.py dependency back into the Makefile.in if that proves troublesome, but it seems to be working locally so far.

Comment 3

[Michael Shal [:mshal]]

Attachment: MozReview Request: Bug 1261283 - allow GENERATED_FILES to write to multiple outputs; r?glandium

Review commit: https://reviewboard.mozilla.org/r/43881/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43881/

Comment 4

[Michael Shal [:mshal]]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bfd0256547ba

Comment 5

[Mike Hommey [:glandium]]

Comment on attachment 8737290 [details]
MozReview Request: Bug 1261283 - allow GENERATED_FILES to write to multiple outputs; r?glandium

https://reviewboard.mozilla.org/r/43881/#review41129

::: python/mozbuild/mozbuild/backend/recursivemake.py:514
(Diff revision 1)
> -            backend_file.write('GARBAGE += %s\n' % obj.output)
> +            backend_file.write('%s:: %s\n' % (tier, first_output))
> +            for output in obj.outputs:
> +                if output != first_output:
> +                    backend_file.write('%s: %s\n' % (output, first_output))

Mmmm ISTR this isn't the right way to do multiple output in Make, but it seems to work in practice. It does however have better messages when invoking the individual targets if you add a ; to the `output: first_output` lines.

Comment 6

[Michael Shal [:mshal]]

(In reply to Mike Hommey [:glandium] from comment #5)
> ::: python/mozbuild/mozbuild/backend/recursivemake.py:514
> (Diff revision 1)
> > -            backend_file.write('GARBAGE += %s\n' % obj.output)
> > +            backend_file.write('%s:: %s\n' % (tier, first_output))
> > +            for output in obj.outputs:
> > +                if output != first_output:
> > +                    backend_file.write('%s: %s\n' % (output, first_output))
> 
> Mmmm ISTR this isn't the right way to do multiple output in Make, but it
> seems to work in practice. It does however have better messages when
> invoking the individual targets if you add a ; to the `output: first_output`
> lines.

According to [1], the main disadvantage of this approach is if you manually remove one of the non-main outputs (xpidlyacc.py in this case), then make won't rebuild it, since xpidllex.py is already up-to-date. This is the approach used previously in the Makefile.in though, so we at least aren't losing functionality here.

I've added the ';' to the end of the 'output, first_output' lines.

[1] https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html

Comment 7

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/305f923c6f9c

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/305f923c6f9c