Closed Bug 967976 Opened 8 years ago Closed 8 years ago

Sometimes building files more than once on mac and linux

Categories

(Release Engineering :: General, defect)

x86_64
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: glandium, Assigned: glandium)

Details

Attachments

(1 file)

See, for example this log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=32767717&tree=Mozilla-Inbound

In that particular build, we're building about 1.6 times the number of files a normal clobber build would. Those are mac universal builds, so all files *are* expected to be built twice, but here's what we get (approximates from a raw grep):
- 1184 files are built four times
- 83 files are built six times
- 145 files are built eight times

So far, this seems to be a mac-only problem.
And it's due to CCACHE_BASEDIR. On my tests on birch, here's the kind of thing i was getting:

Rebuilding Compression.o because /builds/slave/m-in-osx64-nonunified-00000000/build/clang/bin/../lib/clang/3.3/include/stddef.       h, /builds/slave/m-in-osx64-nonunified-00000000/build/clang/bin/../lib/clang/3.3/include/stdint.h, /builds/slave/m-in-osx64-no       nunified-00000000/build/clang/bin/../lib/clang/3.3/include/limits.h was removed

Note the m-in-osx64-nonunified-00000000 path, which is different from bir-osx64-00000000000000000000. So what's happening is that thanks to CCACHE_BASEDIR, a compilation on a random branch can get output from a build on another random branch, which, for objects, is mostly fine, but unfortunately, objects are not the only thing the compiler (and ccache) outputs: dependency files are another. Those dependency files contain path to all files included when preprocessing the source, and that includes all system headers, which, in clang tooltool case, points in the source directory. So when the build gets the dependency file from a build from another branch, it gets one that contains paths in that other branch.

On a clobber build, during the compile tier, dependency files are not around (nor object files), so there's no side effect, but when it's time for the libs tier, the dependency files got there from the compile tier, and since they contain broken dependencies, that makes us rebuild those files.

This might be a ccache bug in how it handles dependency files with dependencies under the BASEDIR. Although, it's weird that it only complains about clang system headers and not about other headers that would be under the BASEDIR.

I think the safest thing to do is to stop setting CCACHE_BASEDIR.
Component: Build Config → General Automation
Product: Core → Release Engineering
QA Contact: catlee
Version: Trunk → unspecified
Attachment #8371218 - Flags: review?(catlee)
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
So, what ccache does with BASEDIR is to rewrite flags, not rewrite the dependency file, so any header under the BASEDIR that would be included by some other things than following what's in -I and other explicit flags is likely to have wrong paths when crossing BASEDIR borders. This happens with clang because of those system headers, I'd actually expect that to happen with tooltool gcc as well, and it so appears that tooltool gcc landed after all the builds i've looked at, so it might well be happening on linux now.

There are two possibilities here, either stop setting CCACHE_BASEDIR, like the patch I attached does, or stop putting the tooltool toolchains under the CCACHE_BASEDIR.
Indeed, this is now happening on linux:
https://tbpl.mozilla.org/php/getParsedLog.php?id=34045221&tree=Mozilla-Inbound

136 files built 4 times, 79 built 3 times, 947 built twice.
Summary: Sometimes building files more than once on mac → Sometimes building files more than once on mac and linux
Attachment #8371218 - Flags: review?(catlee) → review+
in production
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.