1484897 - sccache caches clang profile-use objects

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

This manifests itself intermittently when doing PGO+LTO, with the following error:
[task 2018-08-21T00:56:27.643Z] 00:56:27     INFO -  /builds/worker/workspace/build/src/clang/bin/ld.lld: error: linking module flags 'ProfileSummary': IDs have conflicting values
[task 2018-08-21T00:56:27.643Z] 00:56:27     INFO -  clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)

But this is likely to cause problems when building PGO without LTO too, thus blocking bug 1481721.

This is not a problem on Windows yet because we don't cache anything for other reasons.

Now the question is whether to cherry-pick https://github.com/mozilla/sccache/pull/276 or update sccache to master once it's merged. Ted, what do you think?

Comment 1

[(not currently active) Ted Mielczarek]

There are a lot of changes on master, but 99% of them are for the sccache-dist work, which is behind a cargo feature so it shouldn't impact our builds. It should be fine to just update to master once I merge that.

I'm going to try to get a few more of Aidan's PRs merged first though, since they're large and a pain to rebase. I'm pretty sure one of them touches the clang commandline parsing code.

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1484897 - Update the version of sccache for Linux to support clang PGO

Bug 1476604 tried the same thing for different reasons, but ultimately
failed because of unsupported flags for PGO with clang-cl. However, to
unblock switching to clang for Linux nightlies, we go ahead and upgrade
only the Linux sccache.

Comment 3

[(not currently active) Ted Mielczarek]

This is OK, but I think it's a simple fix to unbreak clang-cl, so it might be nicer to just fix that instead. I just haven't found time to write the small patch necessary.

Comment 4

[Nathan Froyd [:froydnj]]

Comment on attachment 9007519 [details]
Bug 1484897 - Update the version of sccache for Linux to support clang PGO

Nathan Froyd [:froydnj] has approved the revision.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78a680cf3b40
Update the version of sccache for Linux to support clang PGO r=froydnj

Comment 6

[Andrei Ciure[:aciure]]

Backed out changeset 78a680cf3b40 (bug 1484897) for build timeouts

push that caused the backout: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=78a680cf3b404deaa5c95e5ebe7da75cb65a8219&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified

backout: https://hg.mozilla.org/integration/autoland/rev/b9c9088a486865ec1a5275a5a9bd960cfd7e36a1

Comment 7

[Mike Hommey [:glandium]]

Something seems to be going wrong with the new version of sccache.

Comment 8

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/60b4c5197cf2
Backed out changeset 78a680cf3b40 for build timeouts CLOSED TREE

Comment 9

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #7)
> Something seems to be going wrong with the new version of sccache.

It's hard to reproduce. It only seems to happen when there are cache misses (hypothetically*), but also doesn't happen when setting SCCACHE_RECACHE.

* difficulty being added by the fact that failed builds don't have any useful logs (e.g. sccache logs).

Comment 10

[(not currently active) Ted Mielczarek]

(In reply to Mike Hommey [:glandium] from comment #9)
> (In reply to Mike Hommey [:glandium] from comment #7)
> > Something seems to be going wrong with the new version of sccache.

Ugh.

> It's hard to reproduce. It only seems to happen when there are cache misses
> (hypothetically*), but also doesn't happen when setting SCCACHE_RECACHE.
> 
> * difficulty being added by the fact that failed builds don't have any
> useful logs (e.g. sccache logs).

We talked about this before--I think we should just stick sccache.log directly in the upload directory. I looked at doing this a while ago but I got bogged down in all the layers of taskcluster/mozharness stuff. We could probably just do it in a hacky-but-straightforward way.

Comment 11

[(not currently active) Ted Mielczarek]

Well, one obvious thing that we need to fix is:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=70d56b46ec6f221a9693d7404849b12d81bc5ae3&selectedJob=198690806

    sccache requests_not_cacheable opt taskcluster-m5.4xlarge: 3493


from sccache.log:
DEBUG 2018-09-11T17:14:43Z: sccache::server: parse_arguments: CannotCache(-Xclang): ["-std=gnu99", "--target=i686-linux-gnu", "-o", "lz4.o", "-c", "-I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers", "-include", "/builds/worker/workspace/build/src/config/gcc_hidden.h", "-DNDEBUG=1", "-DTRIMMED=1", "-DIMPL_MFBT", "-DLZ4LIB_VISIBILITY=", "-I/builds/worker/workspace/build/src/mfbt", "-I/builds/worker/workspace/build/src/obj-firefox/mfbt", "-I/builds/worker/workspace/build/src/mfbt/double-conversion", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr", "-I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss", "-fPIC", "-include", "/builds/worker/workspace/build/src/obj-firefox/mozilla-config.h", "-DMOZILLA_CLIENT", "-Qunused-arguments", "-U_FORTIFY_SOURCE", "-D_FORTIFY_SOURCE=2", "-march=pentium-m", "-msse", "-msse2", "-mfpmath=sse", "-U_FORTIFY_SOURCE", "-D_FORTIFY_SOURCE=2", "-fno-strict-aliasing", "-ffunction-sections", "-fdata-sections", "-fno-math-errno", "-pthread", "-pipe", "-g", "-Xclang", "-load", "-Xclang", "/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so", "-Xclang", "-add-plugin", "-Xclang", "moz-check", "-O2", "-fno-omit-frame-pointer", "-Werror", "-Qunused-arguments", "-Wall", "-Wempty-body", "-Wignored-qualifiers", "-Wpointer-arith", "-Wshadow-field-in-constructor-modified", "-Wsign-compare", "-Wtype-limits", "-Wunreachable-code", "-Wunreachable-code-return", "-Wclass-varargs", "-Wfloat-overflow-conversion", "-Wfloat-zero-conversion", "-Wloop-analysis", "-Werror=non-literal-null-conversion", "-Wstring-conversion", "-Wtautological-overlap-compare", "-Wno-error=deprecated-declarations", "-Wno-error=array-bounds", "-Wformat", "-Wformat-security", "-Wno-gnu-zero-variadic-macro-arguments", "-MD", "-MP", "-MF", ".deps/lz4.o.pp", "/builds/worker/workspace/build/src/mfbt/lz4.c"]

We have -Xclang marked as TooHard currently, but that's what we're using for the clang plugin:
https://github.com/mozilla/sccache/blob/dda88a658c0f48994527c201e15f736ad193da4f/src/compiler/clang.rs#L89

This was added in:
https://github.com/mozilla/sccache/pull/271

We could either revert that, or we could change how we load clang plugins (bug 1487070).

Comment 12

[Mike Hommey [:glandium]]

Note that on clang-cl builds, we use -Xclang for the equivalent of -MD -MP -MF for dependency files. All in all, we probably need to bite the bullet and add support for -Xclang to sccache.

Comment 13

[(not currently active) Ted Mielczarek]

(In reply to Mike Hommey [:glandium] from comment #12)
> Note that on clang-cl builds, we use -Xclang for the equivalent of -MD -MP
> -MF for dependency files. All in all, we probably need to bite the bullet
> and add support for -Xclang to sccache.

We have support for it in clang-cl:
https://github.com/mozilla/sccache/blob/dda88a658c0f48994527c201e15f736ad193da4f/src/compiler/msvc.rs#L325

There the MSVC commandline parser collects arguments following -Xclang and passes that list to the clang commandline parser, which seems to work OK. The issue here is -Xclang being used to pass cc1 arguments down, which sccache doesn't understand. I guess we could do a similar two-pass parse on those as well, and have a short list of known cc1 arguments.

Comment 14

[(not currently active) Ted Mielczarek]

Per comment 11 I'm wondering if the cause of these timeouts isn't just "sccache isn't caching anything so the compile times are too long and we lowered our timeouts a while ago"?

Comment 15

[(not currently active) Ted Mielczarek]

This was fixed by bug 1476604. The root cause of those timeouts was a bug in tokio-process. We had updated tokio-process on master and wound up with a version with that bug. We updated again to a version that fixed it.