Closed Bug 1357622 Opened 3 years ago Closed 3 years ago

Frequent build failures with "gmake[5]: *** [Unified_cpp_whatever_file.o] Error 254"


(Firefox Build System :: General, defect, critical)

Not set


(firefox-esr52 unaffected, firefox53 unaffected, firefox54 unaffected, firefox55 fixed)

Tracking Status
firefox-esr52 --- unaffected
firefox53 --- unaffected
firefox54 --- unaffected
firefox55 --- fixed


(Reporter: philor, Assigned: ted)



(Keywords: intermittent-failure, Whiteboard: [stockwell fixed])


(1 file)

21 failures in the last 3 days:

:ted, can you look into why we are having build problems so frequenly?  I assume this is related to a recent change.
Flags: needinfo?(ted)
Whiteboard: [stockwell needswork]
This could certainly be a regression from bug 1354048. I will take a look.
Flags: needinfo?(ted)
:ted, following up here, as you can see this is affecting many builds, do you have any updates or need more information?
Flags: needinfo?(ted)
I did a try push with sccache logging enabled, I'll try to reproduce this and hopefully the logs will show what's going on:
Flags: needinfo?(ted)
Ugh, it reproduced but my log upload patch didn't work because when the build fails we don't run the `make upload` stuff!
thanks for continuing to hack on this Ted!
Another attempt:

acrichto also has a PR for sccache that should help make these sorts of issues easier to diagnose in the future:
Assignee: nobody → ted
OK, finally got this to fail with logging enabled:

That build failed compiling Unified_cpp_layout_painting0.o

The sccache log is here:

Searching for Unified_cpp_layout_painting0.o eventually finds:
DEBUG:sccache::server: [Error(Msg("Failed to parse cache entry"), State { next_error: Some(InvalidArchive("Could not find central directory end")) })] compilation failed: "Unified_cpp_layout_painting0.o"

So apparently it failed to open the zip file from the cache? In any event, that shouldn't fail the build, it should just fall back to compiling. I'll get that fixed.

I'll try to see if I can figure out if we're storing bad zip files in s3, or if we're just failing to download the zip files properly.
:ted, do you have any updates here?
Flags: needinfo?(ted)
(In reply to Joel Maher ( :jmaher) from comment #15)
> :ted, do you have any updates here?

Yes, sorry, I have a patch in hand. I will get new sccache binaries built with the fix and do a try push with them today.
Flags: needinfo?(ted)
That try push looks pretty green, except for one Taskcluster Windows debug build, which failed an xpcshell selftest (that seems to be an intermittent failure). Obviously this doesn't prove that it fixes the intermittent build failure here, but after the investigation I did in comment 12 I'm fairly confident that this will fix this specific issue. Additionally, per comment 11, acrichto landed a patch that should give us better errors in the log if a similar situation crops up in the future:
Comment on attachment 8866015 [details]
bug 1357622 - update sccache to rev d3aa1116844b50c03015266d2f48235509fa7deb to pick up a fix for build failures.
Attachment #8866015 - Flags: review?(mshal) → review+
Pushed by
update sccache to rev d3aa1116844b50c03015266d2f48235509fa7deb to pick up a fix for build failures. r=mshal
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Whiteboard: [stockwell needswork] → [stockwell fixed]
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.