Closed Bug 1484897 Opened Last year Closed 7 months ago

sccache caches clang profile-use objects

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: glandium, Assigned: glandium)

References

Details

Attachments

(1 file)

This manifests itself intermittently when doing PGO+LTO, with the following error:
[task 2018-08-21T00:56:27.643Z] 00:56:27     INFO -  /builds/worker/workspace/build/src/clang/bin/ld.lld: error: linking module flags 'ProfileSummary': IDs have conflicting values
[task 2018-08-21T00:56:27.643Z] 00:56:27     INFO -  clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)

But this is likely to cause problems when building PGO without LTO too, thus blocking bug 1481721.

This is not a problem on Windows yet because we don't cache anything for other reasons.

Now the question is whether to cherry-pick https://github.com/mozilla/sccache/pull/276 or update sccache to master once it's merged. Ted, what do you think?
Flags: needinfo?(ted)
There are a lot of changes on master, but 99% of them are for the sccache-dist work, which is behind a cargo feature so it shouldn't impact our builds. It should be fine to just update to master once I merge that.

I'm going to try to get a few more of Aidan's PRs merged first though, since they're large and a pain to rebase. I'm pretty sure one of them touches the clang commandline parsing code.
Flags: needinfo?(ted)
Bug 1476604 tried the same thing for different reasons, but ultimately
failed because of unsupported flags for PGO with clang-cl. However, to
unblock switching to clang for Linux nightlies, we go ahead and upgrade
only the Linux sccache.
Assignee: nobody → mh+mozilla
This is OK, but I think it's a simple fix to unbreak clang-cl, so it might be nicer to just fix that instead. I just haven't found time to write the small patch necessary.
Comment on attachment 9007519 [details]
Bug 1484897 - Update the version of sccache for Linux to support clang PGO

Nathan Froyd [:froydnj] has approved the revision.
Attachment #9007519 - Flags: review+
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/78a680cf3b40
Update the version of sccache for Linux to support clang PGO r=froydnj
Something seems to be going wrong with the new version of sccache.
Flags: needinfo?(ted)
Backout by aciure@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/60b4c5197cf2
Backed out changeset 78a680cf3b40 for build timeouts CLOSED TREE
(In reply to Mike Hommey [:glandium] from comment #7)
> Something seems to be going wrong with the new version of sccache.

It's hard to reproduce. It only seems to happen when there are cache misses (hypothetically*), but also doesn't happen when setting SCCACHE_RECACHE.

* difficulty being added by the fact that failed builds don't have any useful logs (e.g. sccache logs).
(In reply to Mike Hommey [:glandium] from comment #9)
> (In reply to Mike Hommey [:glandium] from comment #7)
> > Something seems to be going wrong with the new version of sccache.

Ugh.

> It's hard to reproduce. It only seems to happen when there are cache misses
> (hypothetically*), but also doesn't happen when setting SCCACHE_RECACHE.
> 
> * difficulty being added by the fact that failed builds don't have any
> useful logs (e.g. sccache logs).

We talked about this before--I think we should just stick sccache.log directly in the upload directory. I looked at doing this a while ago but I got bogged down in all the layers of taskcluster/mozharness stuff. We could probably just do it in a hacky-but-straightforward way.
Flags: needinfo?(ted)
Well, one obvious thing that we need to fix is:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=70d56b46ec6f221a9693d7404849b12d81bc5ae3&selectedJob=198690806

    sccache requests_not_cacheable opt taskcluster-m5.4xlarge: 3493


from sccache.log:
DEBUG 2018-09-11T17:14:43Z: sccache::server: parse_arguments: CannotCache(-Xclang): ["-std=gnu99", "--target=i686-linux-gnu", "-o", "lz4.o", "-c", "-I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers", "-include", "/builds/worker/workspace/build/src/config/gcc_hidden.h", "-DNDEBUG=1", "-DTRIMMED=1", "-DIMPL_MFBT", "-DLZ4LIB_VISIBILITY=", "-I/builds/worker/workspace/build/src/mfbt", "-I/builds/worker/workspace/build/src/obj-firefox/mfbt", "-I/builds/worker/workspace/build/src/mfbt/double-conversion", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss", "-fPIC", "-include", "/builds/worker/workspace/build/src/obj-firefox/mozilla-config.h", "-DMOZILLA_CLIENT", "-Qunused-arguments", "-U_FORTIFY_SOURCE", "-D_FORTIFY_SOURCE=2", "-march=pentium-m", "-msse", "-msse2", "-mfpmath=sse", "-U_FORTIFY_SOURCE", "-D_FORTIFY_SOURCE=2", "-fno-strict-aliasing", "-ffunction-sections", "-fdata-sections", "-fno-math-errno", "-pthread", "-pipe", "-g", "-Xclang", "-load", "-Xclang", "/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so", "-Xclang", "-add-plugin", "-Xclang", "moz-check", "-O2", "-fno-omit-frame-pointer", "-Werror", "-Qunused-arguments", "-Wall", "-Wempty-body", "-Wignored-qualifiers", "-Wpointer-arith", "-Wshadow-field-in-constructor-modified", "-Wsign-compare", "-Wtype-limits", "-Wunreachable-code", "-Wunreachable-code-return", "-Wclass-varargs", "-Wfloat-overflow-conversion", "-Wfloat-zero-conversion", "-Wloop-analysis", "-Werror=non-literal-null-conversion", "-Wstring-conversion", "-Wtautological-overlap-compare", "-Wno-error=deprecated-declarations", "-Wno-error=array-bounds", "-Wformat", "-Wformat-security", "-Wno-gnu-zero-variadic-macro-arguments", "-MD", "-MP", "-MF", ".deps/lz4.o.pp", "/builds/worker/workspace/build/src/mfbt/lz4.c"]

We have -Xclang marked as TooHard currently, but that's what we're using for the clang plugin:
https://github.com/mozilla/sccache/blob/dda88a658c0f48994527c201e15f736ad193da4f/src/compiler/clang.rs#L89

This was added in:
https://github.com/mozilla/sccache/pull/271

We could either revert that, or we could change how we load clang plugins (bug 1487070).
Note that on clang-cl builds, we use -Xclang for the equivalent of -MD -MP -MF for dependency files. All in all, we probably need to bite the bullet and add support for -Xclang to sccache.
(In reply to Mike Hommey [:glandium] from comment #12)
> Note that on clang-cl builds, we use -Xclang for the equivalent of -MD -MP
> -MF for dependency files. All in all, we probably need to bite the bullet
> and add support for -Xclang to sccache.

We have support for it in clang-cl:
https://github.com/mozilla/sccache/blob/dda88a658c0f48994527c201e15f736ad193da4f/src/compiler/msvc.rs#L325

There the MSVC commandline parser collects arguments following -Xclang and passes that list to the clang commandline parser, which seems to work OK. The issue here is -Xclang being used to pass cc1 arguments down, which sccache doesn't understand. I guess we could do a similar two-pass parse on those as well, and have a short list of known cc1 arguments.
Per comment 11 I'm wondering if the cause of these timeouts isn't just "sccache isn't caching anything so the compile times are too long and we lowered our timeouts a while ago"?

This was fixed by bug 1476604. The root cause of those timeouts was a bug in tokio-process. We had updated tokio-process on master and wound up with a version with that bug. We updated again to a version that fixed it.

Depends on: 1476604
Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.