1376593 - sccache makes compiler error messages useless ("cc1plus: error: to generate dependencies you must specify either -M or -MM")

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Masatoshi Kimura [:emk]]

Recently, every time I see a build bustage in automation, I see:
  cc1plus: error: to generate dependencies you must specify either -M or -MM
that is totally useless to find what is wrong. For example, see <https://treeherder.mozilla.org/logviewer.html#?job_id=110023178&repo=try&lineNumber=8000>.

If I'm lucky, clang builds will fail with the same error and I can spot the error:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4b0a70eac8686b5a114f2755854a23f211a5aa35&selectedJob=110023174

If I'm unlucky, it is a really painful guesswork to find the error (for example, bug 1373984 comment #11). It has surprised me that nobody complains about this cryptic error message.

Is this a bug of GCC? If so, we should upgrade GCC in automation. Or is this our fault? If so, what can we do to get more decent messages?

Comment 1

[Masatoshi Kimura [:emk]]

2017-06-28T23:57:08: DEBUG : Starting merge handling...
2017-06-28T23:57:08: DEBUG : Using url: https://hg.mozilla.org/integration/mozilla-inbound/json-pushes?changeset=4e3a5199d4fed80eab8dfd8d06ebe07513dd0587&full=1
2017-06-28T23:57:09: DEBUG : Found commit message:
bug 1357825 - use sccache for caching Rust compilation. r=froydnj

MozReview-Commit-ID: 84PCmiVBlrV

2017-06-28T23:57:09: INFO : The bisection is done.
2017-06-28T23:57:09: INFO : Stopped

Comment 2

[(not currently active) Ted Mielczarek]

We believe this is a regression from https://github.com/mozilla/sccache/commit/0a281848d137bec11322c66b5726d2f508ac8a02 .

Prior to this, if gcc compilation failed with -Werror, we would not retry it. Now we'll retry it, but we're not properly handling some preprocessor arguments (-MP in this specific case), so we're passing gcc a confused set of arguments and it errors.

Comment 3

[Nathan Froyd [:froydnj]]

Ted has this under control!

Comment 4

[Masatoshi Kimura [:emk]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> Prior to this, if gcc compilation failed with -Werror, we would not retry
> it. Now we'll retry it, but we're not properly handling some preprocessor
> arguments (-MP in this specific case), so we're passing gcc a confused set
> of arguments and it errors.

Does it mean the retry is not functioning effectively? (Because the retry will always fail anyway.)
When will this be fixed? If it will take some time, please revert the mozilla/sccache#110 change until the proper retry logic is implemented because the current status is strictly worse than before.

Comment 5

[(not currently active) Ted Mielczarek]

I have a patch:
https://github.com/mozilla/sccache/pull/145

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to Masatoshi Kimura [:emk] from comment #4)
> (In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> > Prior to this, if gcc compilation failed with -Werror, we would not retry
> > it. Now we'll retry it, but we're not properly handling some preprocessor
> > arguments (-MP in this specific case), so we're passing gcc a confused set
> > of arguments and it errors.
> 
> Does it mean the retry is not functioning effectively? (Because the retry
> will always fail anyway.)

There was a bug in the argument handling that made the retry not use the correct set of arguments. This same bug existed in the clang code, but clang doesn't error in this circumstance.

Comment 8

[Masatoshi Kimura [:emk]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #6)
> There was a bug in the argument handling that made the retry not use the
> correct set of arguments. This same bug existed in the clang code, but clang
> doesn't error in this circumstance.

Then please backout <https://hg.mozilla.org/mozilla-central/rev/4e3a5199d4fe> or turn it off at least for gcc ASAP until the sccache bug is fixed.

Comment 10

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=712313262db61385fd9af6822d96acb8c4ead849

Comment 11

[(not currently active) Ted Mielczarek]

Attachment: bug 1376593 - update sccache to 69334a26ba65fc88e3934271a2ce6781c51b445e to fix a regression.

Review commit: https://reviewboard.mozilla.org/r/154596/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/154596/

Comment 12

[Nathan Froyd [:froydnj]]

Comment on attachment 8883686 [details]
bug 1376593 - update sccache to 69334a26ba65fc88e3934271a2ce6781c51b445e to fix a regression.

https://reviewboard.mozilla.org/r/154596/#review159676

Stealing review while gps is out, so this can be landed sooner rather than later.

Comment 13

[(not currently active) Ted Mielczarek]

My apologies, this was just hard to get done in the confines of the work week, and I neglected to finish it off Monday.

Comment 14

[Daniel Holbert [:dholbert]]

[Adding the bogus error message to bug summary to make this bug more discoverable via search / new-bug-form dupe-suggestions]

Comment 15

[(not currently active) Ted Mielczarek]

I had a few broken builds on that try push, but nothing seemed related to the sccache update. I retriggered all the failing builds and the second attempt was green, so I'm going to land this.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4f6450b87658
update sccache to 69334a26ba65fc88e3934271a2ce6781c51b445e to fix a regression. r=froydnj

Comment 17

[(not currently active) Ted Mielczarek]

Sorry about all the annoyance here. I added a test to sccache's test suite that will prevent us from breaking things in this particular way again. Obviously you'll need to update your tree to pull in this changeset to fix this issue in your try pushes.

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/4f6450b87658