sccache makes compiler error messages useless ("cc1plus: error: to generate dependencies you must specify either -M or -MM")

RESOLVED FIXED in Firefox 56

Status

Firefox Build System
General
RESOLVED FIXED
a year ago
5 months ago

People

(Reporter: emk, Assigned: ted)

Tracking

unspecified
mozilla56

Firefox Tracking Flags

(firefox56 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

a year ago
Recently, every time I see a build bustage in automation, I see:
  cc1plus: error: to generate dependencies you must specify either -M or -MM
that is totally useless to find what is wrong. For example, see <https://treeherder.mozilla.org/logviewer.html#?job_id=110023178&repo=try&lineNumber=8000>.

If I'm lucky, clang builds will fail with the same error and I can spot the error:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4b0a70eac8686b5a114f2755854a23f211a5aa35&selectedJob=110023174

If I'm unlucky, it is a really painful guesswork to find the error (for example, bug 1373984 comment #11). It has surprised me that nobody complains about this cryptic error message.

Is this a bug of GCC? If so, we should upgrade GCC in automation. Or is this our fault? If so, what can we do to get more decent messages?
(Reporter)

Comment 1

a year ago
2017-06-28T23:57:08: DEBUG : Starting merge handling...
2017-06-28T23:57:08: DEBUG : Using url: https://hg.mozilla.org/integration/mozilla-inbound/json-pushes?changeset=4e3a5199d4fed80eab8dfd8d06ebe07513dd0587&full=1
2017-06-28T23:57:09: DEBUG : Found commit message:
bug 1357825 - use sccache for caching Rust compilation. r=froydnj

MozReview-Commit-ID: 84PCmiVBlrV

2017-06-28T23:57:09: INFO : The bisection is done.
2017-06-28T23:57:09: INFO : Stopped
Blocks: 1357825
Flags: needinfo?(ted)
Flags: needinfo?(nfroyd)
Summary: GCC (in automation?) generates a useless message for compiler errors → sccache makes compiler error messages useless
We believe this is a regression from https://github.com/mozilla/sccache/commit/0a281848d137bec11322c66b5726d2f508ac8a02 .

Prior to this, if gcc compilation failed with -Werror, we would not retry it. Now we'll retry it, but we're not properly handling some preprocessor arguments (-MP in this specific case), so we're passing gcc a confused set of arguments and it errors.
Assignee: nobody → ted
Flags: needinfo?(ted)
Ted has this under control!
Flags: needinfo?(nfroyd)
(Reporter)

Comment 4

a year ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> Prior to this, if gcc compilation failed with -Werror, we would not retry
> it. Now we'll retry it, but we're not properly handling some preprocessor
> arguments (-MP in this specific case), so we're passing gcc a confused set
> of arguments and it errors.

Does it mean the retry is not functioning effectively? (Because the retry will always fail anyway.)
When will this be fixed? If it will take some time, please revert the mozilla/sccache#110 change until the proper retry logic is implemented because the current status is strictly worse than before.
Flags: needinfo?(ted)
I have a patch:
https://github.com/mozilla/sccache/pull/145
Flags: needinfo?(ted)
(In reply to Masatoshi Kimura [:emk] from comment #4)
> (In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> > Prior to this, if gcc compilation failed with -Werror, we would not retry
> > it. Now we'll retry it, but we're not properly handling some preprocessor
> > arguments (-MP in this specific case), so we're passing gcc a confused set
> > of arguments and it errors.
> 
> Does it mean the retry is not functioning effectively? (Because the retry
> will always fail anyway.)

There was a bug in the argument handling that made the retry not use the correct set of arguments. This same bug existed in the clang code, but clang doesn't error in this circumstance.
(Reporter)

Updated

a year ago
Duplicate of this bug: 1377211
(Reporter)

Comment 8

a year ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #6)
> There was a bug in the argument handling that made the retry not use the
> correct set of arguments. This same bug existed in the clang code, but clang
> doesn't error in this circumstance.

Then please backout <https://hg.mozilla.org/mozilla-central/rev/4e3a5199d4fe> or turn it off at least for gcc ASAP until the sccache bug is fixed.
Flags: needinfo?(ted)
Duplicate of this bug: 1378467
Comment hidden (mozreview-request)

Comment 12

a year ago
mozreview-review
Comment on attachment 8883686 [details]
bug 1376593 - update sccache to 69334a26ba65fc88e3934271a2ce6781c51b445e to fix a regression.

https://reviewboard.mozilla.org/r/154596/#review159676

Stealing review while gps is out, so this can be landed sooner rather than later.
Attachment #8883686 - Flags: review+
My apologies, this was just hard to get done in the confines of the work week, and I neglected to finish it off Monday.
Flags: needinfo?(ted)
[Adding the bogus error message to bug summary to make this bug more discoverable via search / new-bug-form dupe-suggestions]
Summary: sccache makes compiler error messages useless → sccache makes compiler error messages useless ("cc1plus: error: to generate dependencies you must specify either -M or -MM")
I had a few broken builds on that try push, but nothing seemed related to the sccache update. I retriggered all the failing builds and the second attempt was green, so I'm going to land this.
Attachment #8883686 - Flags: review?(gps)

Comment 16

a year ago
Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4f6450b87658
update sccache to 69334a26ba65fc88e3934271a2ce6781c51b445e to fix a regression. r=froydnj
Sorry about all the annoyance here. I added a test to sccache's test suite that will prevent us from breaking things in this particular way again. Obviously you'll need to update your tree to pull in this changeset to fix this issue in your try pushes.

Comment 18

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/4f6450b87658
Status: NEW → RESOLVED
Last Resolved: a year ago
status-firefox56: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla56

Updated

5 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.