build_resources.json missing for windows automation builds

REOPENED
Unassigned

Status

()

Core
Build Config
REOPENED
2 years ago
2 years ago

People

(Reporter: chmanchester, Unassigned)

Tracking

unspecified
mozilla45
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox45 fixed)

Details

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

2 years ago
Not sure if this is a mozharness or build system bug. In http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win64/1447711297/mozilla-inbound-win64-bm85-build1-build2997.txt.gz we have:

15:34:14     INFO - No build_resources.json found, not logging stats

Opt builds appear to be impacted (for whatever reason).
See also bug 893254 and any other open bug filed by me with "psutil" in the summary.

tl;dr Windows builders don't have the ability to compile Python packages with C extensions.

There is also some weirdness with how mach and the build system interact with psutil. configure builds psutil. But mach starts resource monitoring via psutil before it invokes configure. So there is a chicken and egg problem for initial builds.
(Reporter)

Comment 2

2 years ago
Builds on try are building psutil during configure and generating plausible stats (although I can't find them in perfherder, something else to track down). Builds on inbound don't always run configure -- so far it appears we only get psutil when we clobber.

We install psutil from a wheel for all the testers now, this seems like it should be fixable.
(Reporter)

Comment 3

2 years ago
Looks ok on Windows, let's see how other platforms look: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a168628bdcde
(Reporter)

Comment 4

2 years ago
Created attachment 8688720 [details]
MozReview Request: Bug 1225568 - Install psutil from a wheel for windows builds so we can get build stats on dep builds. r=jlund

Bug 1225568 - Intall psutil in scripts/fx_desktop_build.py so we can get build stats on dep builds.

Build stats get logged on try, but only intermittently on inbound. The psutil
extension gets built on windows build machines during configure, but that
step only runs consistently after a clobber. This patch installs psutil from
a wheel in the mozharness virtualenv so it's available in mozharness independent
of what happens in the build.
Attachment #8688720 - Flags: review?(jlund)
(Reporter)

Comment 5

2 years ago
(In reply to Chris Manchester [:chmanchester] from comment #4)
> Created attachment 8688720 [details]
> MozReview Request: Bug 1225568 - Intall psutil in
> scripts/fx_desktop_build.py so we can get build stats on dep builds.
> 
> Bug 1225568 - Intall psutil in scripts/fx_desktop_build.py so we can get
> build stats on dep builds.
> 
> Build stats get logged on try, but only intermittently on inbound. The psutil
> extension gets built on windows build machines during configure, but that
> step only runs consistently after a clobber. This patch installs psutil from
> a wheel in the mozharness virtualenv so it's available in mozharness
> independent
> of what happens in the build.

This is the approach we took in bug 1196393.
(Reporter)

Comment 6

2 years ago
(In reply to Chris Manchester [:chmanchester] from comment #3)
> Looks ok on Windows, let's see how other platforms look:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=a168628bdcde

Has failed spectacularly.
(Reporter)

Updated

2 years ago
Attachment #8688720 - Flags: review?(jlund)
(Reporter)

Updated

2 years ago
Depends on: 1225707
(Reporter)

Comment 7

2 years ago
Created attachment 8689227 [details]
psutil-3.1.1-cp27-none-linux_x86_64.whl
(Reporter)

Comment 8

2 years ago
jlund, can you upload comment 7 to internal pypi when you get a chance (or redirect me appropriately)? Thank you.
Flags: needinfo?(jlund)
We should not be producing wheels with binary content for Linux since distribution of wheels with binaries for Linux has known problems (libc and 3rd party library dependencies aren't properly captured, for example). Source only wheels or wheels with binaries for Mac and Windows are solved problems.

The workaround is to ensure that all Linux machines are able to build Python C extensions and have the build dependencies for these C extensions. I know having a compiler sitting around isn't that awesome. But until Python gets its act together, they leave us little choice.

Note: if our Linux environment is homogeneous, we might be able to get away with wheels with Linux binaries. Just don't do it on PyPI.
(Reporter)

Comment 10

2 years ago
Ok, I built the wheel on a build loaner because that's the one consistent Linux environment we're having this problem. I guess the testers might start picking up this wheel and fail, but if they do we could get mozharness to pass --no-use-wheel when installing psutil.

All our builders can build C extensions, the problem is we don't when we don't run configure on windows. Always installing psutil from a wheel before the build step is just a workaround, limiting the workaround to windows might work too.
The current interaction between mach, virtualenvs, psutil, and the build system is horribly sub-optimal. If you stop by my desk sometime, we can discuss ways to make it suck less. Fixing interaction with psutil is a low hanging fruit in the grand scheme of things.
(Reporter)

Comment 12

2 years ago
Comment on attachment 8689227 [details]
psutil-3.1.1-cp27-none-linux_x86_64.whl

I want to try something else before uploading this.
Attachment #8689227 - Attachment is obsolete: true
Flags: needinfo?(jlund)
(Reporter)

Comment 13

2 years ago
Installing from a wheel on just windows looks fine in https://treeherder.mozilla.org/#/jobs?repo=try&revision=ad5660660fd0

Agreed it's all a mess, but this should help us get more complete stats logged in the short term.
(Reporter)

Comment 14

2 years ago
Comment on attachment 8688720 [details]
MozReview Request: Bug 1225568 - Install psutil from a wheel for windows builds so we can get build stats on dep builds. r=jlund

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/25385/diff/1-2/
Attachment #8688720 - Attachment description: MozReview Request: Bug 1225568 - Intall psutil in scripts/fx_desktop_build.py so we can get build stats on dep builds. → MozReview Request: Bug 1225568 - Install psutil from a wheel for windows builds so we can get build stats on dep builds. r=jlund
Attachment #8688720 - Flags: review?(jlund)
Comment on attachment 8688720 [details]
MozReview Request: Bug 1225568 - Install psutil from a wheel for windows builds so we can get build stats on dep builds. r=jlund

https://reviewboard.mozilla.org/r/25385/#review23061

lgtm

::: testing/mozharness/scripts/fx_desktop_build.py:95
(Diff revision 2)
> +                # building C extensions on linux builders).

as you discovered, these are largely default shared items across all platforms. Since we are defining our own 'virtualenv_modules' in windows only builds, it might be better to remove this default and copy it in the other base configs.

Up to you
Attachment #8688720 - Flags: review?(jlund) → review+
(Reporter)

Comment 16

2 years ago
(In reply to Jordan Lund (:jlund) from comment #15)
> Comment on attachment 8688720 [details]
> MozReview Request: Bug 1225568 - Install psutil from a wheel for windows
> builds so we can get build stats on dep builds. r=jlund
> 
> https://reviewboard.mozilla.org/r/25385/#review23061
> 
> lgtm
> 
> ::: testing/mozharness/scripts/fx_desktop_build.py:95
> (Diff revision 2)
> > +                # building C extensions on linux builders).
> 
> as you discovered, these are largely default shared items across all
> platforms. Since we are defining our own 'virtualenv_modules' in windows
> only builds, it might be better to remove this default and copy it in the
> other base configs.
> 
> Up to you

This sounds like the way to go, but I'd like to do that in a follow up (I'm not entirely sure this is going to fix the entire problem, try only has the successful scenario).

Comment 17

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/39db40982d31
(Reporter)

Comment 18

2 years ago
This appears to have fixed the specific problem of a missing build_resources.json. We're still not getting the whole structure logged on some builds, but I'll continue investigation in bug 1222549.

Comment 19

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/39db40982d31
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox45: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla45
(Reporter)

Comment 20

2 years ago
After more investigation this is still failing intermittently in several ways -- the patch here hasn't helped the situation really.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 21

2 years ago
Backout:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6ba6d14598d0

Comment 22

2 years ago
backoutbugherder
https://hg.mozilla.org/mozilla-central/rev/6ba6d14598d0
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
(Reporter)

Updated

2 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
You need to log in before you can comment on or make changes to this bug.