970567 - make distcc builds work for both HOST and TARGET compilation

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect, P2)

Description

[Dave Huseby [:huseby]]

Currently the TARGET portion of the b2g build uses relative paths to the cross-compiling toolchain.  This prevents distcc from working because it can only handle generic executable calls resolved via search path (e.g. "gcc -c foo.c") and full path executable calls (e.g. "/opt/gcc -c foo.c").  With relative paths, the remote distcc hosts can't find the correct executable to run.

Comment 1

[Dave Huseby [:huseby]]

Attachment: patch to force make to use fully qualified paths to compiler executables when USE_DISTCC is defined

Comment 2

[[:fabrice] Fabrice Desré]

Comment on attachment 8373642 [details] [review]
patch to force make to use fully qualified paths to compiler executables when USE_DISTCC is defined

I'll let mwu and vicamo review.

Comment 3

[Michael Wu [:mwu]]

We should make a TARGET_TOOLCHAIN_ROOT variable and run realpath on that, and then use it to generate TARGET_TOOLS_PREFIX.

Comment 4

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Michael Wu [:mwu] from comment #3)
> We should make a TARGET_TOOLCHAIN_ROOT variable and run realpath on that,
> and then use it to generate TARGET_TOOLS_PREFIX.

But that doesn't work on host toolchain.  For HOST_darwin-x86 and HOST_linux-x86, HOST_{CC,CXX} are aliased to $(CC) and $(CXX) directly.  For HOST_windows-x86, "TOOLS_PREFIX" is used but it's still assigned to empty string by now.  Creating a TARGET_TOOLCHAIN_ROOT, and maybe HOST_TOOLCHAIN_ROOT as well, means you have to touch many many files just like what we had in that pull request.

So why not call $(realpath $($(combo_target)CC)) directly?  "build/core/combo/select.mk" is used to select both TARGET_ and HOST_ mk files.  One can easily get absolute path for those variables by:

  ifneq ($(USE_DISTCC),)
    $(combo_target)CC := $(realpath $($(combo_target)CC))
    $(combo_target)CXX := $(realpath $($(combo_target)CXX))
  endif

in four lines.  Just four lines and you get it for all platforms, both host and target.

Comment 5

[Dave Huseby [:huseby]]

Let me rework this patch to include your suggestions and see if that works.  Thanks for the feedback.

Comment 6

[Dave Huseby [:huseby]]

Comment on attachment 8373642 [details] [review]
patch to force make to use fully qualified paths to compiler executables when USE_DISTCC is defined

I updated the pull request to reflect the feedback.

Comment 7

[Dave Huseby [:huseby]]

I updated the patch to just modify select.mk as suggested in the bugzilla bug. This is all that is needed in the platform_build to make distcc work correctly. The user still needs to have a .userconfig that has something like this in it:

export CC=gcc-4.6
export CXX=g++-4.6
export USE_DISTCC=${USE_DISTCC:-}

# enable distcc if specified
if [ "${USE_DISTCC}" == "1" ]; then
  export CC=`which ${CC}`
  export CXX=`which ${CXX}`
  export CCACHE_PREFIX="distcc"
fi

This will make sure that both CC and CXX have full static path's to the 4.6 compiler on their machine as well as set distcc to be the "prefix command" that ccache uses when calling either compiler. The prefix command settings is essential to making ccache and distcc work well together. 

My results are pretty dramatic. I have a rMBP with a quad-core i7 in it. A full rebuild from scratch on that machine alone takes 45 minutes. I also have a dual quad-core Xeon workstation that I used for distcc and with it in the pool, the full rebuild time drops to 18 mintues. When bisecting bugs in gecko, the combination of ccache and distcc drops the build times to well under 5 minutes. I've seen several builds of only ~1m45s.

Comment 8

[Vicamo Yang [:vicamo][:vyang]]

Comment on attachment 8373642 [details] [review]
patch to force make to use fully qualified paths to compiler executables when USE_DISTCC is defined

Not a Gonk peer, so I place only f+.

Comment 9

[Dave Huseby [:huseby]]

merged: http://bit.ly/MNMXPl

NOTE: this patch does NOT work with the new 4.3 and 4.4 branches of platform_build.  That means it cannot be pulled into the nexus-4 and nexus-4-kk builds as-is.  I was seeing build errors in libicuuc.

Comment 10

[Dave Huseby [:huseby]]

Attachment: a patch to enable distcc for v1.3t tarako builds

the first patch for this bug never made it into v1.3t for tarako builds.  this new patch fixes that.

Comment 11

[Dave Huseby [:huseby]]

re-opening to get the v1.3t patch reviewed and landed.

Comment 12

[Vicamo Yang [:vicamo][:vyang]]

Comment on attachment 8443149 [details] [review]
a patch to enable distcc for v1.3t tarako builds

I think the thing you really need is to ask for 1.3T approval.

Comment 13

[Michael Wu [:mwu]]

Comment on attachment 8443149 [details] [review]
a patch to enable distcc for v1.3t tarako builds

I think it's too late to ask for things on 1.3t unless they're showstoppers.

Comment 14

[Dave Huseby [:huseby]]

alrighty, I'll just keep it as a patch I apply locally.