1665675 - "AttributeError: module distutils has no attribute sysconfig" when building Firefox

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P3)

Description

[Sean Feng [:sefeng]]

This is something that we fixed in bug 1662130, however it came back after bug 1663755.

 0:04.02   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/build/moz.configure/init.configure", line 243, in <module>
 0:04.02     def virtualenv_python3(env_python, virtualenv_name, build_env, mozconfig, help):
 0:04.02   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/python/mozbuild/mozbuild/configure/__init__.py", line 758, in decorator
 0:04.03     depends = DependsFunction(self, func, dependencies, when=when)
 0:04.03   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/python/mozbuild/mozbuild/configure/__init__.py", line 129, in __init__
 0:04.03     sandbox._value_for(self)
 0:04.03   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/python/mozbuild/mozbuild/configure/__init__.py", line 544, in _value_for
 0:04.03     return self._value_for_depends(obj)
 0:04.03   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/python/mozbuild/mozbuild/util.py", line 1021, in method_call
 0:04.03     cache[args] = self.func(instance, *args)
 0:04.03   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/python/mozbuild/mozbuild/configure/__init__.py", line 553, in _value_for_depends
 0:04.03     value = obj.result()
 0:04.03   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/python/mozbuild/mozbuild/util.py", line 1021, in method_call
 0:04.03     cache[args] = self.func(instance, *args)
 0:04.03   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/python/mozbuild/mozbuild/configure/__init__.py", line 155, in result
 0:04.03     return self._func(*resolved_args)
 0:04.03   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/python/mozbuild/mozbuild/configure/__init__.py", line 1164, in wrapped
 0:04.03     return new_func(*args, **kwargs)
 0:04.03   File "/home/sefeng/.local/workspace/mozilla/mozilla-unified4/build/moz.configure/init.configure", line 371, in virtualenv_python3
 0:04.03     if not distutils.sysconfig.get_python_lib():
 0:04.03 AttributeError: module 'distutils' has no attribute 'sysconfig'
 0:04.04 *** Fix above errors and then restart with\
 0:04.04                "./mach build"
 0:04.04 make: *** [client.mk:115: configure] Error 1

I am running Manjaro, a arch based distro.

Comment 1

[Ricky Stewart]

With the latest version of virtualenv I can reproduce this on Ubuntu. I think it's a bug in the configure sandbox.

Comment 2

[Ricky Stewart]

Attachment: Bug 1665675 - Fix sporadic `AttributeError: module distutils has no attribute sysconfig` error in `configure`

The existing implementation of @imports() in the configure sandbox doesn't translate an import of the form @imports('distutils.sysconfig') into an import distutils.sysconfig statement; instead, it transforms the input @imports() request a few times in such a way that we eventually just do import distutils, and expect that distutils.sysconfig will be populated that way. This would be fine, except that this isn't the way that Python's import system works:

>>> import distutils
>>> distutils.sysconfig
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'distutils' has no attribute 'sysconfig'
>>> import distutils.sysconfig
>>> distutils.sysconfig
<module 'distutils.sysconfig' from '/usr/lib/python3.8/distutils/sysconfig.py'>

i.e., we can't just import a parent module and expect that we can indirectly access all child packages of that module without importing them specifically.

So instead, we simplify the current model somewhat by not transforming the @imports() request at all and instead just performing the exact import that the user requested. This resolves the distutils.sysconfig issue as well as hopefully preventing any other similar issues popping up in the future.

While I'm here, I also refactored some stuff so that the way that we're patching in wrapped modules for the sandbox is more structured.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1663755

Comment 4

[Rob Lemley [:rjl]]

D90627 seems to have fixed this issue for me. I'm running Manjaro w/ Python 3.8.5 & viritualenv 20.0.23.

Comment 5

[Rob Wu [:robwu]]

I encountered this problem, and independently reached the same conclusion as written in comment 2.

The @imports('distutils.sysconfig') decorator is incorrectly translated to import distutils as imported.

The patch from comment 2 looks like the desired general solution. Given the constraints I unbroke my build with the following change:

diff --git a/build/moz.configure/init.configure b/build/moz.configure/init.configure
index 970ef849e5e9..bc74d1508c56 100644
--- a/build/moz.configure/init.configure
+++ b/build/moz.configure/init.configure
@@ -233,7 +233,7 @@ option(env='VIRTUALENV_NAME', nargs=1, default='init_py3',
 @imports('os')
 @imports('sys')
 @imports('subprocess')
-@imports('distutils.sysconfig')
+@imports(_from='distutils.sysconfig', _import='get_python_lib')
 @imports(_from='mozbuild.configure.util', _import='LineIO')
 @imports(_from='mozbuild.virtualenv', _import='VirtualenvManager')
 @imports(_from='mozbuild.virtualenv', _import='verify_python_version')
@@ -368,7 +368,7 @@ def virtualenv_python3(env_python, virtualenv_name, build_env, mozconfig, help):
         sys.exit(subprocess.call([python] + sys.argv))
 
     # We are now in the virtualenv
-    if not distutils.sysconfig.get_python_lib():
+    if not get_python_lib():
         die('Could not determine python site packages directory')
 
     str_version = '.'.join(str(v) for v in version)

Comment 6

[fabrixx2]

That patch fixed for me with Arch Linux

Comment 7

[Emily McDonough [:alaskanemily]]

The patch worked for me on VoidLinux.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/37d14279b86b
Fix sporadic `AttributeError: module distutils has no attribute sysconfig` error in `configure` r=glandium

Comment 9

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/37d14279b86b

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

Comment on attachment 9176404 [details]
Bug 1665675 - Fix sporadic AttributeError: module distutils has no attribute sysconfig error in configure

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Resolves configure regression from 1654457
• User impact if declined: Local builds fail during configure
• Fix Landed on Version: 83
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Build patch
• String or UUID changes made by this patch:

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9176404 [details]
Bug 1665675 - Fix sporadic AttributeError: module distutils has no attribute sysconfig error in configure

Needed to better support running mach on newer macOS releases. Approved for 78.8esr.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/90f0978d556c