Closed Bug 1357622 Opened 3 years ago Closed 3 years ago

Frequent build failures with "gmake[5]: *** [Unified_cpp_whatever_file.o] Error 254"

Categories

(Firefox Build System :: General, defect, critical)

defect
Not set
critical

Tracking

(firefox-esr52 unaffected, firefox53 unaffected, firefox54 unaffected, firefox55 fixed)

RESOLVED FIXED
mozilla55
Tracking Status
firefox-esr52 --- unaffected
firefox53 --- unaffected
firefox54 --- unaffected
firefox55 --- fixed

People

(Reporter: philor, Assigned: ted)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell fixed])

Attachments

(1 file)

21 failures in the last 3 days:
https://brasstacks.mozilla.com/orangefactor/index.html?display=Bug&bugid=1357622&startday=2017-04-13&endday=2017-04-20&tree=trunk

:ted, can you look into why we are having build problems so frequenly?  I assume this is related to a recent change.
Flags: needinfo?(ted)
Whiteboard: [stockwell needswork]
This could certainly be a regression from bug 1354048. I will take a look.
Flags: needinfo?(ted)
:ted, following up here, as you can see this is affecting many builds, do you have any updates or need more information?
Flags: needinfo?(ted)
I did a try push with sccache logging enabled, I'll try to reproduce this and hopefully the logs will show what's going on:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec07369811e2800439b7169388bf5dbbcced2512
Flags: needinfo?(ted)
Ugh, it reproduced but my log upload patch didn't work because when the build fails we don't run the `make upload` stuff!
thanks for continuing to hack on this Ted!
Another attempt:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=90ef055457b76169d68fcf5e1330765510259c6e

acrichto also has a PR for sccache that should help make these sorts of issues easier to diagnose in the future:
https://github.com/mozilla/sccache/pull/111
Assignee: nobody → ted
OK, finally got this to fail with logging enabled:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=90ef055457b76169d68fcf5e1330765510259c6e&selectedJob=95897088

That build failed compiling Unified_cpp_layout_painting0.o

The sccache log is here:
https://public-artifacts.taskcluster.net/YrFDLS2BTxaCvaVEYBRdUQ/0/public/build/sccache.log

Searching for Unified_cpp_layout_painting0.o eventually finds:
DEBUG:sccache::server: [Error(Msg("Failed to parse cache entry"), State { next_error: Some(InvalidArchive("Could not find central directory end")) })] compilation failed: "Unified_cpp_layout_painting0.o"

So apparently it failed to open the zip file from the cache? In any event, that shouldn't fail the build, it should just fall back to compiling. I'll get that fixed.

I'll try to see if I can figure out if we're storing bad zip files in s3, or if we're just failing to download the zip files properly.
:ted, do you have any updates here?
Flags: needinfo?(ted)
(In reply to Joel Maher ( :jmaher) from comment #15)
> :ted, do you have any updates here?

Yes, sorry, I have a patch in hand. I will get new sccache binaries built with the fix and do a try push with them today.
Flags: needinfo?(ted)
That try push looks pretty green, except for one Taskcluster Windows debug build, which failed an xpcshell selftest (that seems to be an intermittent failure). Obviously this doesn't prove that it fixes the intermittent build failure here, but after the investigation I did in comment 12 I'm fairly confident that this will fix this specific issue. Additionally, per comment 11, acrichto landed a patch that should give us better errors in the log if a similar situation crops up in the future:
https://github.com/mozilla/sccache/commit/83c14155ae641c180082eff7dfb5cd7745485904
Comment on attachment 8866015 [details]
bug 1357622 - update sccache to rev d3aa1116844b50c03015266d2f48235509fa7deb to pick up a fix for build failures.

https://reviewboard.mozilla.org/r/137612/#review141194
Attachment #8866015 - Flags: review?(mshal) → review+
Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0dcb3c73f936
update sccache to rev d3aa1116844b50c03015266d2f48235509fa7deb to pick up a fix for build failures. r=mshal
https://hg.mozilla.org/mozilla-central/rev/0dcb3c73f936
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Whiteboard: [stockwell needswork] → [stockwell fixed]
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.