1774144 - mach configure fails because of psutils returning nonetype

Status: VERIFIED FIXED

(Firefox Build System :: Mach Core, defect, P1)

Description

[Landry Breuil (:gaston)]

mach configure log is:

Configure complete!
Be sure to run |mach build| to pick up any changes
Error running mach:
    ['configure']
The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file configure| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.
If filing a bug, please include the full output of mach, including this error
message.
The details of the failure are as follows:
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
  File "/build/buildslave-amd64/mozilla-central-amd64/build/build/mach_initialize.py", line 451, in post_dispatch_handler
    _finalize_telemetry_glean(
  File "/build/buildslave-amd64/mozilla-central-amd64/build/build/mach_initialize.py", line 544, in _finalize_telemetry_glean
    system_metrics.physical_cores.add(physical_cores)
  File "/var/buildslave/.mozbuild/srcdirs/build-33ee10420e9d/_virtualenvs/mach/lib/python3.9/site-packages/glean/_uniffi.py", line 1761, in add
    amount = int(amount)

i've tried disabling glean/telemetry with this ~/.machrc but that doesnt help:

[mach_telemetry]
is_enabled = false
[build]
telemetry = false

see http://buildbot.rhaalovely.net/nine/#/builders/3/builds/1371 and previous build for complete failing logs

Comment 1

[Landry Breuil (:gaston)]

last i looked, i blamed it on psutils not returning anything for cpu_count(logical=False) on OpenBSD:

>>> psutil.cpu_count(logical=False)
>>> psutil.cpu_count()
8

Comment 2

[Alessio Placitelli [:Dexter]]

Thank you for trusting us and filing this, :gaston, following up on our Matrix conversation.

Jan-Erik, this seems related to the recent UniFFI change. Would you kindly double check?

More in general, I'm wondering if we should rather record an error instead of failing hard in these cases.

Comment 3

[Jan-Erik Rediger [:janerik]]

More of an integration bug than a Glean bug, but I have a patch ready.

Comment 4

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1774144 - mach: Only report physical cores if they could be determined.

As per the docs:

Return the number of logical CPUs in the system (same as os.cpu_count in Python 3.4) or None if undetermined.
[...] On OpenBSD and NetBSD psutil.cpu_count(logical=False) always return None

Glean tries to coerce values to the expected type. None cannot be
coerced to an integer, so this blows up.
We conditionally report that value now only if we can properly detect
one.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Mach Core' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 6

[Landry Breuil (:gaston)]

Comment on attachment 9281133 [details]
Bug 1774144 - mach: Only report physical cores if they could be determined.

Fwiw this definitely helps, ./mach configure with the patch locally applied succeeds now on OpenBSD:

 1:15.89  0:02.97 File already read. Skipping: /build/buildslave-amd64/mozilla-central-amd64/build/intl/components/moz.build
 1:19.09  0:06.17 File already read. Skipping: /build/buildslave-amd64/mozilla-central-amd64/build/gfx/angle/targets/angle_common/moz.build
 2:12.18 Finished reading 2032 moz.build files in 12.94s
 2:12.18 Read 60 gyp files in parallel contributing 9.65s to total wall time
 2:12.18 Processed into 13354 build config descriptors in 17.38s
 2:12.18 RecursiveMake backend executed in 15.89s
 2:12.18   4444 total backend files; 0 created; 0 updated; 4444 unchanged; 0 deleted; 28 -> 1646 Makefile
 2:12.18 FasterMake backend executed in 1.80s
 2:12.18   12 total backend files; 0 created; 0 updated; 12 unchanged; 0 deleted
 2:12.18 Total wall time: 59.24s; CPU time: 48.64s; Efficiency: 82%; Untracked: 11.24s
Configure complete!
Be sure to run |mach build| to pick up any changes
c64$ 

many thanks for the superquick fix! I should have figured it out myself but that would have taken much more time..

Comment 7

[Jan-Erik Rediger [:janerik]]

(In reply to Landry Breuil (:gaston) from comment #6)

Comment on attachment 9281133 [details]
Bug 1774144 - mach: Only report physical cores if they could be determined.

Fwiw this definitely helps, ./mach configure with the patch locally applied succeeds now on OpenBSD:
[...]
many thanks for the superquick fix! I should have figured it out myself but that would have taken much more time..

Thanks for confirming this!

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/58fef18812b8
mach: Only report physical cores if they could be determined. r=firefox-build-system-reviewers,ahochheiden

Comment 9

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/58fef18812b8

Comment 10

[Landry Breuil (:gaston)]

can confirm that m-c builds fine for me, my build from last night was fine (cf http://buildbot.rhaalovely.net/nine/#/builders/3/builds/1373) - it failed to package but that's because of diskfull on the builder :)