1645393 - build_resources.html is missing graphs

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Nathan Froyd [:froydnj]]

I don't understand how this happened, but the build_resource.html files from builds from:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=c7dbcbcbb07745ad124640bc639c86a251c66300

and following no longer display nice graphs, e.g. Linux64 opt:

https://firefoxci.taskcluster-artifacts.net/Fj_6T0uGSkyZQniaw0vOlQ/0/public/build/build_resources.html

Compare the previous push:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=fab7c4f54054ceb06504c7ddac380858e2521fc4

and the Linux64 opt build_resources.html:

https://firefoxci.taskcluster-artifacts.net/HFwprFgRQpO9_R4797RirA/0/public/build/build_resources.html

But I don't see anything obvious in https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=c7dbcbcbb07745ad124640bc639c86a251c66300 unless there were changes to taskcluster or something that altered how we take perf stats? Individual phases in the no-graph build_resources.json don't have any useful sub-stats:

https://firefoxci.taskcluster-artifacts.net/Fj_6T0uGSkyZQniaw0vOlQ/0/public/build/build_resources.json

so clearly something has gone wrong.

Comment 1

[Nathan Froyd [:froydnj]]

sfraser, did we change anything between those two pushes in infra that would be relevant?

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 3

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Nathan Froyd [:froydnj] from comment #1)

sfraser, did we change anything between those two pushes in infra that would be relevant?

I had https://phabricator.services.mozilla.com/D77472 but that's not landed, as we swapped to integrating this sort of data collection into the worker (which also hasn't landed yet).

It doesn't look like SystemResourceMonitor hasn't changed since 2019.

The symptoms look like what we'd expect if psutil wasn't installed. There's missing information under the system key, and the zeroed out data is probably from the stub psutil that gets used if import psutil fails: https://searchfox.org/mozilla-central/source/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py#19

There are some warnings about Pipenv, though I don't know if those would affect a psutil install. There's certainly at least one successful psutil install in there, but there are many virtualenvs.

[task 2020-06-11T04:44:32.271Z] 04:44:32  WARNING -  check> Warning: Your Pipfile requires python_version 2.7, but you are using 3.6.3 (/builds/worker/w/o/_/o/bin/python).
[task 2020-06-11T04:44:32.271Z] 04:44:32     INFO -  check>   $ pipenv check will surely fail.
[task 2020-06-11T04:44:32.271Z] 04:44:32  WARNING -  check> Warning: Your Pipfile requires python_version 2.7, but you are using 3.6.3 (/builds/worker/w/o/_/o/bin/python).
[task 2020-06-11T04:44:32.271Z] 04:44:32     INFO -  check>   $ pipenv check will surely fail.

Comment 4

[Mike Hommey [:glandium]]

There's this during the configure run:

Error processing command. Ignoring because optional. (optional:setup.py:third_party/python/psutil:build_ext:--inplace)

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1621960

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1645393 - Install python3 development files in build docker images.

It's required to properly setup psutil in the python3 virtualenv, and
psutil is required for build resources tracking by mach.

It's not clear to me why this hasn't been a problem since mach build
switched to python3, though. ¯_(ツ)_/¯

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78830d19b507
Install python3 development files in build docker images. r=firefox-build-system-reviewers,rstewart

Comment 8

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/78830d19b507