1162594 - build_resources.json not getting logged in some circumstances

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Michael Shal [:mshal]]

Bug 1159838 started logging build_resources.json to influxdb, but some builds don't find the file for some reason. In particular, we haven't posted any stats from any Windows builds yet, and none of the mozilla-central builds worked either. I'm not sure yet if mozharness is looking in the wrong place for the file, or if mach isn't generating it in certain cases, or maybe it's being generated and then deleted before we grab it.

Comment 1

[Michael Shal [:mshal]]

I dug into this a little bit - this appears to be because the 'import psutil' is failing in resourcemonitor.py:

13:50:20     INFO -    File "/builds/slave/try-l64-0000000000000000000000/build/src/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 14, in <module>
13:50:20     INFO -      import psutil
13:50:20     INFO -    File "/builds/slave/try-l64-0000000000000000000000/build/src/python/psutil/psutil/__init__.py", line 89, in <module>
13:50:20     INFO -      import psutil._pslinux as _psplatform
13:50:20     INFO -    File "/builds/slave/try-l64-0000000000000000000000/build/src/python/psutil/psutil/_pslinux.py", line 20, in <module>
13:50:20     INFO -      from psutil import _common
13:50:20     INFO -  ImportError: cannot import name _common

(We don't normally print this exception - the build ignores the error and just proceeds without stat collection so as not to break things over stats).

Bug 893254 covers deploying psutil to more places, which should get us the mach stats for every build, I think.

Comment 2

[Michael Shal [:mshal]]

Attachment: 0001-Bug-1162594-create-build_resources.json-even-if-psut.patch

I'd like to record at least the duration of the various mach tiers even if we don't have psutil. This just stubs out the psutil functions if it fails to import, but we still get a build_resources.json at the end of the build with durations that can be logged to influx.

Comment 3

[Michael Shal [:mshal]]

Comment on attachment 8617682 [details] [diff] [review]
0001-Bug-1162594-create-build_resources.json-even-if-psut.patch

Strangely this works on Linux & Windows, but not Mac. I need to look into it some more...

Comment 4

[Michael Shal [:mshal]]

Comment on attachment 8617682 [details] [diff] [review]
0001-Bug-1162594-create-build_resources.json-even-if-psut.patch

This part is fine - just need a mozharness patch to make sure it looks in the right place for the file.

Comment 5

[Michael Shal [:mshal]]

Attachment: buildresources

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8620553 [details] [diff] [review]
buildresources

Review of attachment 8620553 [details] [diff] [review]:
-----------------------------------------------------------------

moar config items! ;)

Comment 7

[Andrew Halberstadt [:ahal]]

Comment on attachment 8617682 [details] [diff] [review]
0001-Bug-1162594-create-build_resources.json-even-if-psut.patch

Review of attachment 8617682 [details] [diff] [review]:
-----------------------------------------------------------------

This looks safe enough and I'm ok with it going into mozbase, so r+.. but gps originally wrote this code and I'm not familiar with psutil.. so take that r+ with a grain of salt. If you feel this warrants a more proper review I suggest flagging him.

Comment 8

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/86016bd9da3b

Comment 9

[Michael Shal [:mshal]]

Comment on attachment 8620553 [details] [diff] [review]
buildresources

https://hg.mozilla.org/build/mozharness/rev/a5a777d6cf06

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/86016bd9da3b