Closed Bug 1484897 Opened Last year Closed 7 months ago
sccache caches clang profile-use objects
46 bytes, text/x-phabricator-request
|Details | Review|
This manifests itself intermittently when doing PGO+LTO, with the following error: [task 2018-08-21T00:56:27.643Z] 00:56:27 INFO - /builds/worker/workspace/build/src/clang/bin/ld.lld: error: linking module flags 'ProfileSummary': IDs have conflicting values [task 2018-08-21T00:56:27.643Z] 00:56:27 INFO - clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation) But this is likely to cause problems when building PGO without LTO too, thus blocking bug 1481721. This is not a problem on Windows yet because we don't cache anything for other reasons. Now the question is whether to cherry-pick https://github.com/mozilla/sccache/pull/276 or update sccache to master once it's merged. Ted, what do you think?
There are a lot of changes on master, but 99% of them are for the sccache-dist work, which is behind a cargo feature so it shouldn't impact our builds. It should be fine to just update to master once I merge that. I'm going to try to get a few more of Aidan's PRs merged first though, since they're large and a pain to rebase. I'm pretty sure one of them touches the clang commandline parsing code.
Bug 1476604 tried the same thing for different reasons, but ultimately failed because of unsupported flags for PGO with clang-cl. However, to unblock switching to clang for Linux nightlies, we go ahead and upgrade only the Linux sccache.
This is OK, but I think it's a simple fix to unbreak clang-cl, so it might be nicer to just fix that instead. I just haven't found time to write the small patch necessary.
Comment on attachment 9007519 [details] Bug 1484897 - Update the version of sccache for Linux to support clang PGO Nathan Froyd [:froydnj] has approved the revision.
Attachment #9007519 - Flags: review+
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/78a680cf3b40 Update the version of sccache for Linux to support clang PGO r=froydnj
Backed out changeset 78a680cf3b40 (bug 1484897) for build timeouts push that caused the backout: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=78a680cf3b404deaa5c95e5ebe7da75cb65a8219&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified backout: https://hg.mozilla.org/integration/autoland/rev/b9c9088a486865ec1a5275a5a9bd960cfd7e36a1
Something seems to be going wrong with the new version of sccache.
Backout by firstname.lastname@example.org: https://hg.mozilla.org/integration/autoland/rev/60b4c5197cf2 Backed out changeset 78a680cf3b40 for build timeouts CLOSED TREE
(In reply to Mike Hommey [:glandium] from comment #7) > Something seems to be going wrong with the new version of sccache. It's hard to reproduce. It only seems to happen when there are cache misses (hypothetically*), but also doesn't happen when setting SCCACHE_RECACHE. * difficulty being added by the fact that failed builds don't have any useful logs (e.g. sccache logs).
(In reply to Mike Hommey [:glandium] from comment #9) > (In reply to Mike Hommey [:glandium] from comment #7) > > Something seems to be going wrong with the new version of sccache. Ugh. > It's hard to reproduce. It only seems to happen when there are cache misses > (hypothetically*), but also doesn't happen when setting SCCACHE_RECACHE. > > * difficulty being added by the fact that failed builds don't have any > useful logs (e.g. sccache logs). We talked about this before--I think we should just stick sccache.log directly in the upload directory. I looked at doing this a while ago but I got bogged down in all the layers of taskcluster/mozharness stuff. We could probably just do it in a hacky-but-straightforward way.
Well, one obvious thing that we need to fix is: https://treeherder.mozilla.org/#/jobs?repo=try&revision=70d56b46ec6f221a9693d7404849b12d81bc5ae3&selectedJob=198690806 sccache requests_not_cacheable opt taskcluster-m5.4xlarge: 3493 from sccache.log: DEBUG 2018-09-11T17:14:43Z: sccache::server: parse_arguments: CannotCache(-Xclang): ["-std=gnu99", "--target=i686-linux-gnu", "-o", "lz4.o", "-c", "-I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers", "-include", "/builds/worker/workspace/build/src/config/gcc_hidden.h", "-DNDEBUG=1", "-DTRIMMED=1", "-DIMPL_MFBT", "-DLZ4LIB_VISIBILITY=", "-I/builds/worker/workspace/build/src/mfbt", "-I/builds/worker/workspace/build/src/obj-firefox/mfbt", "-I/builds/worker/workspace/build/src/mfbt/double-conversion", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss", "-fPIC", "-include", "/builds/worker/workspace/build/src/obj-firefox/mozilla-config.h", "-DMOZILLA_CLIENT", "-Qunused-arguments", "-U_FORTIFY_SOURCE", "-D_FORTIFY_SOURCE=2", "-march=pentium-m", "-msse", "-msse2", "-mfpmath=sse", "-U_FORTIFY_SOURCE", "-D_FORTIFY_SOURCE=2", "-fno-strict-aliasing", "-ffunction-sections", "-fdata-sections", "-fno-math-errno", "-pthread", "-pipe", "-g", "-Xclang", "-load", "-Xclang", "/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so", "-Xclang", "-add-plugin", "-Xclang", "moz-check", "-O2", "-fno-omit-frame-pointer", "-Werror", "-Qunused-arguments", "-Wall", "-Wempty-body", "-Wignored-qualifiers", "-Wpointer-arith", "-Wshadow-field-in-constructor-modified", "-Wsign-compare", "-Wtype-limits", "-Wunreachable-code", "-Wunreachable-code-return", "-Wclass-varargs", "-Wfloat-overflow-conversion", "-Wfloat-zero-conversion", "-Wloop-analysis", "-Werror=non-literal-null-conversion", "-Wstring-conversion", "-Wtautological-overlap-compare", "-Wno-error=deprecated-declarations", "-Wno-error=array-bounds", "-Wformat", "-Wformat-security", "-Wno-gnu-zero-variadic-macro-arguments", "-MD", "-MP", "-MF", ".deps/lz4.o.pp", "/builds/worker/workspace/build/src/mfbt/lz4.c"] We have -Xclang marked as TooHard currently, but that's what we're using for the clang plugin: https://github.com/mozilla/sccache/blob/dda88a658c0f48994527c201e15f736ad193da4f/src/compiler/clang.rs#L89 This was added in: https://github.com/mozilla/sccache/pull/271 We could either revert that, or we could change how we load clang plugins (bug 1487070).
Note that on clang-cl builds, we use -Xclang for the equivalent of -MD -MP -MF for dependency files. All in all, we probably need to bite the bullet and add support for -Xclang to sccache.
(In reply to Mike Hommey [:glandium] from comment #12) > Note that on clang-cl builds, we use -Xclang for the equivalent of -MD -MP > -MF for dependency files. All in all, we probably need to bite the bullet > and add support for -Xclang to sccache. We have support for it in clang-cl: https://github.com/mozilla/sccache/blob/dda88a658c0f48994527c201e15f736ad193da4f/src/compiler/msvc.rs#L325 There the MSVC commandline parser collects arguments following -Xclang and passes that list to the clang commandline parser, which seems to work OK. The issue here is -Xclang being used to pass cc1 arguments down, which sccache doesn't understand. I guess we could do a similar two-pass parse on those as well, and have a short list of known cc1 arguments.
Per comment 11 I'm wondering if the cause of these timeouts isn't just "sccache isn't caching anything so the compile times are too long and we lowered our timeouts a while ago"?
Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.