967976 - Sometimes building files more than once on mac and linux

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mike Hommey [:glandium]]

See, for example this log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=32767717&tree=Mozilla-Inbound

In that particular build, we're building about 1.6 times the number of files a normal clobber build would. Those are mac universal builds, so all files *are* expected to be built twice, but here's what we get (approximates from a raw grep):
- 1184 files are built four times
- 83 files are built six times
- 145 files are built eight times

So far, this seems to be a mac-only problem.

Comment 1

[Mike Hommey [:glandium]]

And it's due to CCACHE_BASEDIR. On my tests on birch, here's the kind of thing i was getting:

Rebuilding Compression.o because /builds/slave/m-in-osx64-nonunified-00000000/build/clang/bin/../lib/clang/3.3/include/stddef.       h, /builds/slave/m-in-osx64-nonunified-00000000/build/clang/bin/../lib/clang/3.3/include/stdint.h, /builds/slave/m-in-osx64-no       nunified-00000000/build/clang/bin/../lib/clang/3.3/include/limits.h was removed

Note the m-in-osx64-nonunified-00000000 path, which is different from bir-osx64-00000000000000000000. So what's happening is that thanks to CCACHE_BASEDIR, a compilation on a random branch can get output from a build on another random branch, which, for objects, is mostly fine, but unfortunately, objects are not the only thing the compiler (and ccache) outputs: dependency files are another. Those dependency files contain path to all files included when preprocessing the source, and that includes all system headers, which, in clang tooltool case, points in the source directory. So when the build gets the dependency file from a build from another branch, it gets one that contains paths in that other branch.

On a clobber build, during the compile tier, dependency files are not around (nor object files), so there's no side effect, but when it's time for the libs tier, the dependency files got there from the compile tier, and since they contain broken dependencies, that makes us rebuild those files.

This might be a ccache bug in how it handles dependency files with dependencies under the BASEDIR. Although, it's weird that it only complains about clang system headers and not about other headers that would be under the BASEDIR.

I think the safest thing to do is to stop setting CCACHE_BASEDIR.

Comment 2

[Mike Hommey [:glandium]]

Attachment: Stop setting CCACHE_BASEDIR

Comment 3

[Mike Hommey [:glandium]]

So, what ccache does with BASEDIR is to rewrite flags, not rewrite the dependency file, so any header under the BASEDIR that would be included by some other things than following what's in -I and other explicit flags is likely to have wrong paths when crossing BASEDIR borders. This happens with clang because of those system headers, I'd actually expect that to happen with tooltool gcc as well, and it so appears that tooltool gcc landed after all the builds i've looked at, so it might well be happening on linux now.

There are two possibilities here, either stop setting CCACHE_BASEDIR, like the patch I attached does, or stop putting the tooltool toolchains under the CCACHE_BASEDIR.

Comment 4

[Mike Hommey [:glandium]]

Indeed, this is now happening on linux:
https://tbpl.mozilla.org/php/getParsedLog.php?id=34045221&tree=Mozilla-Inbound

136 files built 4 times, 79 built 3 times, 947 built twice.

Comment 5

[Mike Hommey [:glandium]]

https://hg.mozilla.org/build/buildbotcustom/rev/88cf0e711c72

Comment 6

[John Hopkins (:jhopkins)]

in production