1966570 - C-C thunderbird fails with "ModuleNotFoundError: No module named 'pathlib._local'; 'pathlib' is not a package"

Status: NEW

(Firefox Build System :: MozillaBuild, defect, P3)

Description

[ISHIKAWA, Chiaki]

I am producing and submitting patches for C-C TB from time to time.
Lately, there was an issue with TSAN-related bug and thus I build TSAN-enabled C-C TB locally.
It built until the middle of April. (I checked and I found a local test log that took place on April 13.)

However, lately the build fails with the following error:
I am quoting a bit more context before the error message.

/home/ishikawa/.mozbuild/srcdirs/mozilla-0332c0b6111a/_virtualenvs/build/bin/python -m mozbuild.action.file_generate /NEW-SSD/NREF-COMM-CENTRAL/mozilla/toolkit/components/telemetry/build_scripts/gen_scalar_data.py generate_JSON_definitions toolkit/components/telemetry/ScalarArtifactDefinitions.json toolkit/components/telemetry/.deps/ScalarArtifactDefinitions.json.pp toolkit/components/telemetry/.deps/ScalarArtifactDefinitions.json.stub /NEW-SSD/NREF-COMM-CENTRAL/mozilla/toolkit/components/telemetry/Scalars.yaml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozbuild/mozbuild/action/file_generate.py", line 154, in <module>
    sys.exit(main(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozbuild/mozbuild/action/file_generate.py", line 98, in main
    ret = module.__dict__[method](
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/toolkit/components/glean/build_scripts/glean_parser_ext/interesting_serverknobs.py", line 15, in main
    all_objs, options = parse(args)
                        ^^^^^^^^^^^
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/toolkit/components/glean/build_scripts/glean_parser_ext/run_glean_parser.py", line 97, in parse
    cached_objects, cached_options = pickle.load(cache)
                                     ^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pathlib._local'; 'pathlib' is not a package
./dependentlibs.list.stub

I am not quite familiar with the code and I almost gave up and decided to let TSAN job handled on treeherder.
But it is not quite convenient.

I have found a workaround which I post in my next comment.

Comment 1

[ISHIKAWA, Chiaki]

When the build failure was noticed and I searched around for a solution the following URL did not exist.
It is three weeks old.

https://github.com/ultralytics/yolov5/issues/13578

Please see the particular comment.
https://github.com/ultralytics/yolov5/issues/13578#issuecomment-2833282100
It says, add the following to our code.

You can add:

import os
import sys
import pathlib

sys.modules["pathlib._local"] = pathlib 
if os.name == 'nt':
   pathlib.PosixPath = pathlib.WindowsPath
else:
   pathlib.WindowsPath = pathlib.PosixPath

before your code

OK, so I modified the code, run_glean_parser.py as follows.

hg diff
diff --git a/toolkit/components/glean/build_scripts/glean_parser_ext/run_glean_parser.py b/toolkit/components/glean/build_scripts/glean_parser_ext/run_glean_parser.py
--- a/toolkit/components/glean/build_scripts/glean_parser_ext/run_glean_parser.py
+++ b/toolkit/components/glean/build_scripts/glean_parser_ext/run_glean_parser.py
@@ -337,9 +337,15 @@ def jog_factory(output_fd, *args):
 
 def jog_file(output_fd, *args):
     all_objs, options = parse(args)
     jog.output_file(all_objs, output_fd, options)
     return get_deps()
 
 
 if __name__ == "__main__":
+    sys.modules["pathlib._local"] = pathlib
+    if os.name == 'nt':
+        pathlib.PosixPath = pathlib.WindowsPath
+    else:
+        pathlib.WindowsPath = pathlib.PosixPath
+
     main(sys.stdout, *sys.argv[1:])

Voila!

The build for TSAN now works. (Strange thing is that my ordinary ASAN build worked without this change, and for that matter, ordinary build using GCC instead of CLANG also worked. Only the TSAN build failed with the particular error I report here.)

This is under Debian GNU/Linux.
I think I am using a particular version of Python because Debian's standard package was a bit too advanced for a proper operation and thus I was forced to pin down the particular version of python maybe around the end of the year or the new year.
I have the following code in my script to build C-C TB TSAN version (pinning the python version using PYENV).
This may explain the behaviour on my PC if other don't see this on their computers.
But the existing Q&A in the quoted https://github.com/ultralytics/yolov5/issues/13578#issuecomment-2833282100
suggests the problem may be with new Python for others as well.

echo setting PYENV_VERSION
export PYENV_VERSION=3.11.11

python --version

I wonder if someone in the know can shed light on this matter and I wonder if there are better fixes.
If not, I can push the patch using moz-phab.

One other thing that confused me is that I can still invoke TSAN build and TSAN tests on treeherder. There must be a version difference somewhere but I cannot figure it out.

Comment 2

[ISHIKAWA, Chiaki]

Debian's python (I am using trixie and testing repositories) on my PC is

$ python --version
Python 3.13.3

Comment 3

[Alex Hochheiden [:ahochheiden] [he/him]]

It's unclear to me from your post, when did you change which version of Python you were using? I'm assuming this is some problem with the .cached being pickled on an older Python, now failing on 3.13 (or the other way around?). I think if you delete the cache it'll work.

If I am right, it's still a problem. It shouldn't matter if you change your Python version, it should still work and do the necessary cache invalidation. Your proposed fix is probably not the correct way to resolve this, but I'll leave it to Serge since it appears he implemented this.

Comment 4

[ISHIKAWA, Chiaki]

(In reply to Alex Hochheiden [:ahochheiden] from comment #3)

It's unclear to me from your post, when did you change which version of Python you were using? I'm assuming this is some problem with the .cached being pickled on an older Python, now failing on 3.13 (or the other way around?). I think if you delete the cache it'll work.

If I am right, it's still a problem. It shouldn't matter if you change your Python version, it should still work and do the necessary cache invalidation. Your proposed fix is probably not the correct way to resolve this, but I'll leave it to Serge since it appears he implemented this.

Thank you for the comment.

I pinned down the Python version probably early January this year. I checked local archive. I already have this change (pinning down the particular version) in my script on Jan 11 while I did not have the change in November 2024.

Strange thing is I could build TSAN version with the changed setup (pinning down) until about the middle of April. I had a local test log of TSAN run on April 13. So I could build a TSAN version until that time. (So the .cache worked?)
But the build somehow began to fail with the error mentioned in this bug until yesterday when I noticed the post from someone who experienced the same issue and had a suggested workaround.

I recall trying to remove .cache directory, but come to think of it, since now I am pinning down the version of Python, I probably needed to remove the cache for the pinned down version(?). Also, there is a virtual environment created during the build / test of C-C TB. I simply gave up not knowing exactly where I should be clearing the cache.
But yesterday's patch I posted solved the issue.

Another mysterious thing is that it was only TSAN build which failed.
Ordinary build using gcc instead of clang worked. ASAN build with clang also worked.
Only TSAN build with clang started to fail in mid-April until the posted patch was applied.
I would think all these builds are similarly affected. Oh wait, have I pinned down the python version for the other build scripts?
I checked and all these scripts have the same idiom to use the pinned down version of python.

It would be great to have the scripts hardened to invalidate the cache when it is necessary if the stale cache was the reason for error.

Too bad, I can't recall why I needed to use a particular version of Python in early January. There was a compatibility issue, to be sure.
Otherwise, I won't bother to change scripts to pin down the version of Python for C-C TB build.

Comment 5

[Takanori MATSUURA]

Hi Chiaki,
Forgive me if this comes off as preachy.

I have been following your work on Mozilla Bugzilla for a long time, and I respect your contributions. However, your build environment is often very localized. Consequently, problems that occur in your environment are sometimes not reproduced by other users. This can be problematic when trying to determine if the problem exists in your local environment or Mozilla's source tree.

First, you should try to minimize localization.
Your local patches should be reflected in comm-central and mozilla-central via Bugzilla as soon as possible to reduce your maintenance costs and frustration for other Mozilla developers who see your reports. This benefits both you and other Mozilla developers.

Comment 6

[ISHIKAWA, Chiaki]

(In reply to Takanori MATSUURA from comment #5)

Hi Chiaki,
Forgive me if this comes off as preachy.

I have been following your work on Mozilla Bugzilla for a long time, and I respect your contributions. However, your build environment is often very localized. Consequently, problems that occur in your environment are sometimes not reproduced by other users. This can be problematic when trying to determine if the problem exists in your local environment or Mozilla's source tree.

First, you should try to minimize localization.
Your local patches should be reflected in comm-central and mozilla-central via Bugzilla as soon as possible to reduce your maintenance costs and frustration for other Mozilla developers who see your reports. This benefits both you and other Mozilla developers.

Takanori san,

Sorry to respond so late.
I was sidetracked in the 2nd half of last year due to various engagements.
Yes my local environment has fortified itself after many customization over the years to the point I am not sure what has happened to it.
Each minor customization dealt with binary tool chain issues (C compiler, rust version for TSAN build, minor and major issues to run C-C TB under valgrind), and various package dependencies under Debian GNU/Linux.
I am trying to offload the issue by contributing back the local changes that were necessitated by system issues that happened over the last dozen years or so.

TSAN build was broken on my local PC, when somehow I accidentally updated the version to a later version not meant for TSAN. I could not figure out the problem and now about a couple of months ago, I reinstalled rustc for TSAN and carefully pinned down the version for it because ordinary build and ASAN seem to be happy with the later rustc version AND the particular version meant for TSAN.
It is complicated :-(

I admire the admins and developers who make it possible for us to run jobs including ASAN, TSAN, etc.
It must be very tough to maintain the correct environment for all these jobs.
However, I do realize that each built environment on treeherder is created afresh for submitted jobs. In a sense that makes it easy.
I have other software development environment on the same Debian GNU/Linux and I cannot easily dictate the version of various packages to suit C-C TB's needs only :-(

BTW, rustc version is pinned down like this now.

pushd ~
~ /NREF-COMM-CENTRAL/mozilla /NREF-COMM-CENTRAL/mozilla/comm
ishikawa@ip030:~$ rustc --version
rustc 1.93.1 (01f6ddf75 2026-02-11)   <--- ordinary programs use this version.
ishikawa@ip030:~$ pushd 
/NREF-COMM-CENTRAL/mozilla ~ /NREF-COMM-CENTRAL/mozilla/comm
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ rustc --version
rustc 1.90.0-dev (1159e78c4 2025-09-14)    <---- TSAN build needs this particular version
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ 

Maybe I should create a container for C-C TB build locally starting from scratch.