1662130 - "AttributeError: module distutils has no attribute sysconfig" when building Firefox

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

I'm not hitting this personally, but there was some chatter on #developers where at least two developers are seeing it.

The error is happening here:
https://searchfox.org/mozilla-central/rev/969fc7fa6c3c7fc489f53b7b7f8c902028b5169f/build/moz.configure/init.configure#382

I believe it's happening because some Linux distros patch the distutils module, and it looks like at least ArchLinux (and possibly also Debian) don't have that sysconfig attribute.

In general distutils is pretty unreliable (I think it's even possible for it to be missing entirely from certain distributions).

If we can't find an alternative way to find that info, I wonder if we could solve this by fixing bug 1622789. Maybe it would be possible to vendor the canonical version of distutils?

Comment 1

[Ricky Stewart]

I wonder what the reference to distutils.sysconfig is even doing there. We don't use get_python_lib() there for anything. I assume it can be deleted. Of course there are more references to the module in-tree and I don't know which can and cannot be safely replaced with other stuff.

Comment 2

[Ricky Stewart]

Historically we haven't had a problem with depending on distutils.sysconfig outright. On Ubuntu there's a Debian package to install it. Are the reporters just people running Ubuntu who haven't installed it?

Comment 3

[Andrew Halberstadt [:ahal]]

They are CC'ed, :pbz at least mentioned it was on ArchLinux.

Comment 4

[Emma Zühlcke [:emz]]

(In reply to Ricky Stewart from comment #2)

Historically we haven't had a problem with depending on distutils.sysconfig outright. On Ubuntu there's a Debian package to install it. Are the reporters just people running Ubuntu who haven't installed it?

I'd expect mach bootstrap to set that up for me. Or is that the wrong assumption?

(In reply to Andrew Halberstadt [:ahal] from comment #3)

They are CC'ed, :pbz at least mentioned it was on ArchLinux.

Yes, I'm running Arch. Is there a package I need to install?

Comment 5

[Ricky Stewart]

No, not to my knowledge. This is a bug of some sort, but I don't know where it's coming from.

I asked you this in Riot, but to be clear: Does locally reverting bug 1660351 (hg revision f50e4bee2efb) fix the issue?

Comment 6

[Ricky Stewart]

I'd expect mach bootstrap to set that up for me. Or is that the wrong assumption?

Yeah, bootstrap does not necessarily make your Python environment "correct" for you. The UX around this is a work-in-progress. But I don't think that's the root cause of your problem.

Comment 7

[Ricky Stewart]

Oh. Also, Paul, can you post a complete (unabridged) log from mach configure demonstrating the failure?

Comment 8

[Ricky Stewart]

Attachment: Bug 1662130 - Remove `inherit-from-parent-environment` logic from `virtualenv` handling

This logic is meant to expose packages from a globally-installed Python to be used by the in-objdir virtualenvs, so for example we don't have to figure out how to install zstandard (or other Python packages with native code that may or may not have prebuilt wheels for any given platform) in those virtualenvs. This worked mostly, but is causing builds to unconditionally break on Arch Linux, caused a couple test failures, and in general is just introducing other weird behaviors downstream, and issues with the resultant PYTHONPATHs are hard to diagnose and fix.

In the long-term we'll have to permanently solve the zstandard problem and pave the way for other Python packages with native code as well, but that's not an urgent need.

Partially reverts bugs 1660351 and 1656993. Entirely reverts bug 1660353, restoring that file to as it was before that patch.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1660351

Comment 10

[Ricky Stewart]

Attachment: Bug 1662130 - Walk back new `inherit-from-parent-environment` logic in `virtualenv` handling

This logic is meant to expose packages from a globally-installed Python to be used by the in-objdir virtualenvs, so for example we don't have to figure out how to install zstandard (or other Python packages with native code that may or may not have prebuilt wheels for any given platform) in those virtualenvs. Bug 1660351 augmented that logic to work within the requirements of bug 1660353. This worked mostly, but is causing builds to unconditionally break on Arch Linux, caused a couple test failures, and in general is just introducing other weird behaviors downstream, and issues with the resultant PYTHONPATHs are hard to diagnose and fix.

In the long-term we'll have to permanently solve the zstandard problem and pave the way for other Python packages with native code as well, but that's not an urgent need.

The ultimate goal is to completely remove inherit-from-parent-environment, but we can't do that until bug 1659539 is solved.

Partially reverts bugs 1660351. Entirely reverts bug 1660353, restoring that file to as it was before that patch.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3ac7571a9498
Walk back new `inherit-from-parent-environment` logic in `virtualenv` handling r=ahal

Comment 13

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/3ac7571a9498

Comment 14

[Sean Feng [:sefeng211]]

I am hitting this again, filed bug 1665675 for it.