1859676 - Port bug 1857516: Build fails on Python 3.12; the module `imp` is now removed

Status: RESOLVED FIXED

(Thunderbird :: Upstream Synchronization, defect)

Description

[Rob Lemley [:rjl]]

The upstream change causes an error when comm-central is present.

  File "/home/rob/moz/m-c/comm/build/mach_initialize.py", line 59, in initialize
    from mach.command_util import MACH_COMMANDS, MachCommandReference
  File "/home/rob/moz/m-c/python/mach/mach/command_util.py", line 15, in <module>
    from mozfile import load_source
ModuleNotFoundError: No module named 'mozfile'

This is due to the insanity of comm/build/mach_initialize.py -- the mach virtualenv cannot be activated prior to calling m-c's mach initialize function. But, we need to use parts of mach.command_util to set up Thunderbird mach commands before calling mach initialize...

mach.command_util now depends on mozfile, so it's path needs to be added to sys.path earlier.

This is all to avoid a tot of code duplication...

Comment 1

[Rob Lemley [:rjl]]

Attachment: Bug 1859676 - Port bug 1857516: Add mozfile to tb_common.txt mach site. r=#thunderbird-reviewers

It's needed to import mach.command_util now. The mach site cannot be initialzed
twice.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/79ed9674160a
Port bug 1857516: Add mozfile to tb_common.txt mach site. r=darktrojan

Comment 4

[Kai Engert [:KaiE:]]

Rob, despite this fix, I'm still affected, see also my report in bug 1859797.

I get:
ImportError: cannot import name 'load_source' from 'mozfile' (/home/user/.local/lib/python3.11/site-packages/mozfile/init.py)

Comment 5

[Rob Lemley [:rjl]]

Is there a reason you have mozfile installed in the user-site-packages directory?

That copy is apparently old, as the "load_source" function was added to the in-tree copy in bug 1857516 just two days ago.

Quick fix is to either remove mozfile from your site-packages directory so the in-tree copy is used, or you could update your local copy from the one in m-c:testing/mozbase/mozfile.

Personally I think it's kind of annoying that load_source wound up in mozfile in the first place... it really should be in mach.util which doesn't require having a mach virtualenv in place already... I'll file another bug for that and see where it goes...

Comment 6

[Rob Lemley [:rjl]]

Attachment: Bug 1859676 - Follow-up: Insert mozfile path at beginning of sys.path to always use in-tree copy. r=kaie

Comment 7

[Kai Engert [:KaiE:]]

(In reply to Rob Lemley [:rjl] from comment #5)

Is there a reason you have mozfile installed in the user-site-packages directory?

I cannot say. I don't think I ever manually installed it. Could I have gotten it in the past by executing "mach bootstrap", or as part of other automatic actions performed by "mach"?

Are you certain I indeed have this mozfile thing installed on my system?

Do you want me to remove directory $HOME/.local/lib/python3.11/site-packages/mozfile - either by removing it, or by using some other uninstall method?

Comment 8

[Kai Engert [:KaiE:]]

I've tested, if I move away directory .local/lib/python3.11/site-packages/mozfile - then the build works fine.

FWIW, the files inside that directory are from April 2023, not very old.

For completeness, I will restore the directory, and try your other suggestions, too.

Comment 9

[Kai Engert [:KaiE:]]

I did some investigation on my local system.

I found several other python modules on my system, which were installed at the same time (the files have the same timestamp).

It looks like they all originate from installing the "mozregression".

(I had installed that in my usual development environment, without any isolation.)

Comment 10

[Rob Lemley [:rjl]]

(In reply to Kai Engert (:KaiE:) from comment #9)

I did some investigation on my local system.

I found several other python modules on my system, which were installed at the same time (the files have the same timestamp).

It looks like they all originate from installing the "mozregression".

(I had installed that in my usual development environment, without any isolation.)

Ahh yeah, I install that into an isolated virtualenv...

The posted followup should work for your situation, and I've filed bug 1860114 to deal with this problem upstream. I should have a patch for that ready later today (I hope). Then the extra comm-central hackery is reduced to something a little less insane.

Comment 11

[Kai Engert [:KaiE:]]

Thanks Rob, you helped me a lot.

For me it was important that the problem is understood what was going on and to find a solution, and you have done that!

For my local system, I'm ok to go with the simple solution, remove mozfile and uninstall mozregression, I never actually used them on this system, and in the future I'll install such tools in isolation, so I don't need the attached patch to land, if you prefer the general solution upstream.

Comment 12

[Rob Lemley [:rjl]]

I'm going to land just in case someone else has the same issue.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/3ee6b987d296
Follow-up: Insert mozfile path at beginning of sys.path to always use in-tree copy. r=kaie

Comment 15

[Rob Lemley [:rjl]]

Bug 1860114 is a WONTFIX, so this is now resolved FIXED since it is working.