1718878 - Python 3.10 Support: Use `collections.abc` instead of `collections` for collection abstract base class usages

Status: RESOLVED FIXED

(Firefox Build System :: General, task, P3)

Description

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0

Steps to reproduce:

Update the local hg repository for nightly in fedora rawhide
Try to run mach build

Actual results:

mach failed to start because did not find 'Iterables' in 'collections' when it tried to import it.
Traceback (most recent call last):
  File "/mnt/to_archive/accum/src/firefox-source/./mach", line 171, in <module>
    main(sys.argv[1:])
  File "/mnt/to_archive/accum/src/firefox-source/./mach", line 163, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "/mnt/to_archive/accum/src/firefox-source/./mach", line 150, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/mnt/to_archive/accum/src/firefox-source/./mach", line 138, in load_mach
    return mach_bootstrap.bootstrap(dir_path)
  File "/mnt/to_archive/accum/src/firefox-source/build/mach_bootstrap.py", line 205, in bootstrap
    import mach.main
  File "/mnt/to_archive/accum/src/firefox-source/python/mach/mach/main.py", line 19, in <module>
    from collections import Iterable
ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib64/python3.10/collections/__init__.py)

Expected results:

mach runs

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::General' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[[e-mail]]

It looks like the collections module was removed in python 3.10 after being deprecated since python 3.3. There is a new module called collections.abc, and when I run the command in the text interface, it succeeds. So, maybe this fix for python 3.10 is as simple as changing collections to collections.abc.

Comment 3

[[e-mail]]

I can still import collections in 3.10, and it still contains OrderedDict, but Iterable is only in collections.abc. Each use of collections will have to be examined to determine whether it needs to be changed or not.

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Makes sense to me, thanks for reporting this 👍
Do you want to try making a patch for this? If you haven't contributed a patch before, there's some docs here.

I can still import collections in 3.10, and it still contains OrderedDict, but Iterable is only in collections.abc. Each use of collections will have to be examined to determine whether it needs to be changed or not.

The approach here should be: if it's possible to import the type from collections.abc, then do so (it's an abstract collection type). Otherwise, continue importing from collections (the rest are concrete types, e.g. ChainMap).

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

I'm going to modify this ticket to be about converting the remaining necessary collections -> collections.abc, since there's already a patch up to fix the global mach failure.

Comment 6

[[e-mail]]

Thanks for your help. I thought about creating a patch until I saw the extent of the problem (from memory, 45 fixes just for collections). I would be fixing things without a clear understanding of the logic behind the problem. I decided to wait until I saw your (knowledgeable, expert) response. Looking at the main bug, https://bugzilla.mozilla.org/show_bug.cgi?id=1719144, I've decided that this is better left to the experts. I'll keep the reference above in bookmarks, though, just in case. :-)

Comment 7

[[e-mail]]

I fixed all these problems in a new cloned hg repository. And successfully built nightly. But, I see that the fix is already in in the other bugzilla, so will not be submitting a patch. There were problems in the python/, testing/, and third_party/ directories.

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

This good-first-bug hasn't had any activity for 2 months, it is automatically unassigned.
For more information, please visit auto_nag documentation.

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

I'm pulling this off the good-first-bug list to earmark this for an onboarding build engineer (Hi! 👋)

As documented in Python 3.9 for the collections module:

Deprecated since version 3.3, will be removed in version 3.10: Moved Collections Abstract Base Classes to the collections.abc module. For backwards compatibility, they continue to be visible in this module through Python 3.9.

Python 3.10 is being released in a few weeks, but there's quite a few places where we are incorrectly still using collections instead of collections.abc.

What you'll need to do is:

• For each of the collections imports, check if it's importing a Collection Abstract Base Class. If yes, modify it accordingly.
  • For example, Iterable should be from collections.abc, but defaultdict must still be imported from collections.
• Don't modify code that's either in the "Third-party code" section on Searchfox, or within testing/web-platform/tests (which is also essentially third-party code).

Comment 10

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1718878 - Minimal changes to third_party library gyp to make it compatible with Python 3.10 r?mhentges

gyp appears to have been abandoned. To make it compatible with Python 3.10 a
small 'soft fork' had to be made. If gyp is ever updated in the future, these
changes can probably be discarded.

gyp source here: https://chromium.googlesource.com/external/gyp/

Comment 11

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1718878 - Updates to the build system to enable compatibility with Python 3.10 (Mainly regarding collections.abc) r?mhentges

Previously, collections.abc was available from the collections import. In Python
3.10, they were explicitly moved to collections.abc. As such, any references to
collections abstract base classes need to import collections.abc in Python 3.10.

More reading here: https://docs.python.org/3/library/collections.abc.html

Depends on D128837

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2dd19e8c9cec
Minimal changes to third_party library gyp to make it compatible with Python 3.10 r=mhentges
https://hg.mozilla.org/integration/autoland/rev/734e384a08d2
Updates to the build system to enable compatibility with Python 3.10 (Mainly regarding collections.abc) r=mhentges,webdriver-reviewers,whimboo

Comment 13

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/2dd19e8c9cec
https://hg.mozilla.org/mozilla-central/rev/734e384a08d2

Comment 14

[Rob Lemley [:rjl]]

Could this get uplifted to esr91? Building with Python 3.10.2 fails: AttributeError: module 'collections' has no attribute 'Iterable'.

Comment 15

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Comment on attachment 9246572 [details]
Bug 1718878 - Minimal changes to third_party library gyp to make it compatible with Python 3.10 r?mhentges

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Allow building Firefox with Python 3.10
• User impact if declined: Firefox/Thunderbird won't be able to be built with modern Python
• Fix Landed on Version: 95
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Not risky because if it failed it would be the build, not at run-time.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9246572 [details]
Bug 1718878 - Minimal changes to third_party library gyp to make it compatible with Python 3.10 r?mhentges

Approved for ESR91.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr91/rev/ffe34cdc6f44
https://hg.mozilla.org/releases/mozilla-esr91/rev/3b175cd808ef