Linux builds on cedar failing to find build dependencies

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jgraham, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
https://tbpl.mozilla.org/?tree=Cedar&rev=2fafa4415c2e

One linux build failed at:

/bin/sh: make: command not found

A Linux 64 build failed with:

/builds/slave/ced-l64-0000000000000000000000/build/source/client.mk:316: *** Could not find autoconf 2.13.  Stop.
(Reporter)

Comment 1

3 years ago
Oh, and the code seems to build locally under mach.
I would not be surprised if the machines were not able to get to tooltool (where a lot of those tools that we download from it).
Unfortunately, the logs do not say that it failed.
I've triggered a 3rd Linux opt to see if it repeats.
(Reporter)

Comment 3

3 years ago
That retrigger seems to have failed in the same way.
I think we might have been running mozharness fx desktop on Cedar and we have now been regressed.

jlund: would you mind helping us disable fx desktop mozharness? Thanks!

Comment 5

3 years ago
yes it seems like things have bit-rotted. /me adds this to buildduty queue. I'll resolve by EOD

Comment 6

3 years ago
it turns out this wasn't bit rot. This mozharness patch broke it: http://hg.mozilla.org/build/mozharness/rev/b2069bb5736e

Basically I was calling run_command_m() directly instead of using enable_mock() and disable_mock() in between run_command()

creating a patch now to fix.

Comment 7

3 years ago
Created attachment 8441857 [details] [diff] [review]
140617_bug_1026468_cedar_linux_builds_stopped_using_mock.patch

see https://bugzilla.mozilla.org/show_bug.cgi?id=1026468#c6 for the reason of this patch.

implementation details:
    the paradigm for b2g builds usually gets us to enable_mock() and disable_mock() once or twice for a script run and most jobs use mock. for desktop builds, we go back and forth a lot between using and not using mock and many of our platforms don't use it at all.
    For those reasons, I am creating a _possibly_run_command_with_mock() method so I do not repeat myself and I am explicit about what I am doing.
Attachment #8441857 - Flags: review?(sphink)

Updated

3 years ago
See Also: → bug 1027146
Comment on attachment 8441857 [details] [diff] [review]
140617_bug_1026468_cedar_linux_builds_stopped_using_mock.patch

Review of attachment 8441857 [details] [diff] [review]:
-----------------------------------------------------------------

r=me for now, but I think we'll want to merge buildbase.py and buildb2gbase.py (unlanded; part of bug 898554), and that uses a separate config (b2g_build.py passes in a gecko_config) to choose the mock config (including the mock_target). But I think that just means we'll want to look at this again and figure out how to make it all work as cleanly as possible.

Oh. Another way that duplicating this logic will cause trouble for the merge -- buildb2gbase.py has a --disable-mock flag that I know you also wanted here, and checks it during enable_mock. That actually wouldn't break with this patch because (1) they're separate inheritance hierarchies atm anyway, so you'd never call both; and (2) _possibly... would just let the call go through to enable_mock, which would do nothing.

Anyway, good enough for now. Sorry for the bustage.
Attachment #8441857 - Flags: review?(sphink) → review+

Comment 9

3 years ago
this didn't land. we opted to backout original cause of bustage for now and come up with a more robust solution later: https://bugzilla.mozilla.org/show_bug.cgi?id=1027146#c8

backout is live in prod and this should be resolved. Let's re-trigger some linux jobs to confirm before resolving. That is after the tree closure lifts :)
(Reporter)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.