Closed Bug 985837 Opened 11 years ago Closed 10 years ago

emulator-kk builds crash make half the time (which it describes as "failed to build")

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
All
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Unassigned)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files)

force -j4 for emulator-kk
1.46 KB, patch
mozilla
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
force -j1 for emulator-kk
1.22 KB, patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
Like https://tbpl.mozilla.org/php/getParsedLog.php?id=36420777&tree=Mozilla-Central. I'd find it to be a build improvement, just like it said on the tin, except people keep retriggering them until they turn green, instead of accepting the blessing of a quick red. "Maybe we can force build.sh to use -j <10."
this is definitely make crashing, as opposed to bash or nice: Mar 20 03:54:56 bld-linux64-spot-332 kernel: make[12579]: segfault at 0 ip 0000000000416f61 sp 00007fff78be61c0 error 4 in make[400000+28000] Mar 20 03:54:56 bld-linux64-spot-332 abrt[24810]: Can't read /proc/12579/exe link
we've got make-3.81-19.el6.x86_64 installed
Hidden on m-i, m-c and b-i, since they couldn't see the visiblity requirement of <5% failure if they climbed a ladder.
And hidden on mozilla-aurora, but it seems I'm back on the &onlyunstarred=1 so rest assured I'll keep you updated on how frequently they fail.
build.sh doesn't actually set -j anything. It looks like .config has: MAKE_FLAGS=-j10 And config.sh has this in it: echo MAKE_FLAGS=-j$((CORE_COUNT + 2)) > .tmp-config So the simplest way of overriding this would be to do: echo "MAKE_FLAGS=-j4" >> .config after running config.sh and before running build.sh Otherwise we could add an option to config.sh that allows the default -j setting to be set when config.sh is run.
no idea if this will help or not, but worth a shot
Attachment #8396860 - Flags: review?(aki)
Comment on attachment 8396860 [details] [diff] [review] force -j4 for emulator-kk Thanks Catlee.
Attachment #8396860 - Flags: review?(aki) → review+
Attachment #8396860 - Flags: checked-in+
That seems to have taken care of the "half the time" problem: now they fail to build 100% of the time.
huzzah let's try -j1 then
Attachment #8397819 - Flags: review?(rail)
Attachment #8397819 - Flags: review?(rail) → review+
Attachment #8397819 - Flags: checked-in+
in production
in production
Okay, almost sort of builds fairly often now, except that it hits bug 934890 an awful lot of the time,
Still hidden, still a constant source of failure, but not from this :)
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
FTR backed out this change since it seems to be working well in staging without the workaround.
Should be a fun bit of statistical analysis, figuring out whether the (still continuing) 14400 second timeouts have decreased enough to make the rate of these worth tolerating.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Something here went live today
make 3.82 seems to work...in staging
Chris, what are next steps here? :-)
Flags: needinfo?(catlee)
Depends on: 1017166
Blocks: 1017545
KK builds hidden on all trees for too many failures; will unhide in bug 1017545, once bug 1017166 fixes the failures here.
Flags: needinfo?(catlee)
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: