1225568 - build_resources.json missing for windows automation builds

Status: REOPENED

(Firefox Build System :: General, defect)

Description

[Chris Manchester (limited bugmail, email directly)]

Not sure if this is a mozharness or build system bug. In http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win64/1447711297/mozilla-inbound-win64-bm85-build1-build2997.txt.gz we have:

15:34:14     INFO - No build_resources.json found, not logging stats

Opt builds appear to be impacted (for whatever reason).

Comment 1

[Gregory Szorc [:gps]]

See also bug 893254 and any other open bug filed by me with "psutil" in the summary.

tl;dr Windows builders don't have the ability to compile Python packages with C extensions.

There is also some weirdness with how mach and the build system interact with psutil. configure builds psutil. But mach starts resource monitoring via psutil before it invokes configure. So there is a chicken and egg problem for initial builds.

Comment 2

[Chris Manchester (limited bugmail, email directly)]

Builds on try are building psutil during configure and generating plausible stats (although I can't find them in perfherder, something else to track down). Builds on inbound don't always run configure -- so far it appears we only get psutil when we clobber.

We install psutil from a wheel for all the testers now, this seems like it should be fixable.

Comment 3

[Chris Manchester (limited bugmail, email directly)]

Looks ok on Windows, let's see how other platforms look: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a168628bdcde

Comment 4

[Chris Manchester (limited bugmail, email directly)]

Attachment: MozReview Request: Bug 1225568 - Install psutil from a wheel for windows builds so we can get build stats on dep builds. r=jlund

Bug 1225568 - Intall psutil in scripts/fx_desktop_build.py so we can get build stats on dep builds.

Build stats get logged on try, but only intermittently on inbound. The psutil
extension gets built on windows build machines during configure, but that
step only runs consistently after a clobber. This patch installs psutil from
a wheel in the mozharness virtualenv so it's available in mozharness independent
of what happens in the build.

Comment 5

[Chris Manchester (limited bugmail, email directly)]

(In reply to Chris Manchester [:chmanchester] from comment #4)
> Created attachment 8688720 [details]
> MozReview Request: Bug 1225568 - Intall psutil in
> scripts/fx_desktop_build.py so we can get build stats on dep builds.
> 
> Bug 1225568 - Intall psutil in scripts/fx_desktop_build.py so we can get
> build stats on dep builds.
> 
> Build stats get logged on try, but only intermittently on inbound. The psutil
> extension gets built on windows build machines during configure, but that
> step only runs consistently after a clobber. This patch installs psutil from
> a wheel in the mozharness virtualenv so it's available in mozharness
> independent
> of what happens in the build.

This is the approach we took in bug 1196393.

Comment 6

[Chris Manchester (limited bugmail, email directly)]

(In reply to Chris Manchester [:chmanchester] from comment #3)
> Looks ok on Windows, let's see how other platforms look:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=a168628bdcde

Has failed spectacularly.

Comment 7

[Chris Manchester (limited bugmail, email directly)]

Attachment: psutil-3.1.1-cp27-none-linux_x86_64.whl

Comment 8

[Chris Manchester (limited bugmail, email directly)]

jlund, can you upload comment 7 to internal pypi when you get a chance (or redirect me appropriately)? Thank you.

Comment 9

[Gregory Szorc [:gps]]

We should not be producing wheels with binary content for Linux since distribution of wheels with binaries for Linux has known problems (libc and 3rd party library dependencies aren't properly captured, for example). Source only wheels or wheels with binaries for Mac and Windows are solved problems.

The workaround is to ensure that all Linux machines are able to build Python C extensions and have the build dependencies for these C extensions. I know having a compiler sitting around isn't that awesome. But until Python gets its act together, they leave us little choice.

Note: if our Linux environment is homogeneous, we might be able to get away with wheels with Linux binaries. Just don't do it on PyPI.

Comment 10

[Chris Manchester (limited bugmail, email directly)]

Ok, I built the wheel on a build loaner because that's the one consistent Linux environment we're having this problem. I guess the testers might start picking up this wheel and fail, but if they do we could get mozharness to pass --no-use-wheel when installing psutil.

All our builders can build C extensions, the problem is we don't when we don't run configure on windows. Always installing psutil from a wheel before the build step is just a workaround, limiting the workaround to windows might work too.

Comment 11

[Gregory Szorc [:gps]]

The current interaction between mach, virtualenvs, psutil, and the build system is horribly sub-optimal. If you stop by my desk sometime, we can discuss ways to make it suck less. Fixing interaction with psutil is a low hanging fruit in the grand scheme of things.

Comment 12

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8689227 [details]
psutil-3.1.1-cp27-none-linux_x86_64.whl

I want to try something else before uploading this.

Comment 13

[Chris Manchester (limited bugmail, email directly)]

Installing from a wheel on just windows looks fine in https://treeherder.mozilla.org/#/jobs?repo=try&revision=ad5660660fd0

Agreed it's all a mess, but this should help us get more complete stats logged in the short term.

Comment 14

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8688720 [details]
MozReview Request: Bug 1225568 - Install psutil from a wheel for windows builds so we can get build stats on dep builds. r=jlund

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/25385/diff/1-2/

Comment 15

[Jordan Lund (:jlund)]

Comment on attachment 8688720 [details]
MozReview Request: Bug 1225568 - Install psutil from a wheel for windows builds so we can get build stats on dep builds. r=jlund

https://reviewboard.mozilla.org/r/25385/#review23061

lgtm

::: testing/mozharness/scripts/fx_desktop_build.py:95
(Diff revision 2)
> +                # building C extensions on linux builders).

as you discovered, these are largely default shared items across all platforms. Since we are defining our own 'virtualenv_modules' in windows only builds, it might be better to remove this default and copy it in the other base configs.

Up to you

Comment 16

[Chris Manchester (limited bugmail, email directly)]

(In reply to Jordan Lund (:jlund) from comment #15)
> Comment on attachment 8688720 [details]
> MozReview Request: Bug 1225568 - Install psutil from a wheel for windows
> builds so we can get build stats on dep builds. r=jlund
> 
> https://reviewboard.mozilla.org/r/25385/#review23061
> 
> lgtm
> 
> ::: testing/mozharness/scripts/fx_desktop_build.py:95
> (Diff revision 2)
> > +                # building C extensions on linux builders).
> 
> as you discovered, these are largely default shared items across all
> platforms. Since we are defining our own 'virtualenv_modules' in windows
> only builds, it might be better to remove this default and copy it in the
> other base configs.
> 
> Up to you

This sounds like the way to go, but I'd like to do that in a follow up (I'm not entirely sure this is going to fix the entire problem, try only has the successful scenario).

Comment 17

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/39db40982d31

Comment 18

[Chris Manchester (limited bugmail, email directly)]

This appears to have fixed the specific problem of a missing build_resources.json. We're still not getting the whole structure logged on some builds, but I'll continue investigation in bug 1222549.

Comment 19

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/39db40982d31

Comment 20

[Chris Manchester (limited bugmail, email directly)]

After more investigation this is still failing intermittently in several ways -- the patch here hasn't helped the situation really.

Comment 21

[Pulsebot]

Backout:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6ba6d14598d0

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/6ba6d14598d0