1769631 - python-3.11 compatibility

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement, P3)

Description

[Joonas Niilola]

Attachment: firefox-100.0.1-build.log

Steps to reproduce:

Try to build firefox-100.0.1 using python-3.11.

Actual results:

Error running mach:

    ['configure']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file configure| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ValueError: invalid mode: 'rU'

  File "/var/tmp/portage/www-client/firefox-100.0.1/work/firefox-100.0.1/python/mozbuild/mozbuild/build_commands.py", line 255, in configure
    return driver.configure(
           ^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/www-client/firefox-100.0.1/work/firefox-100.0.1/python/mozbuild/mozbuild/controller/building.py", line 1648, in configure
    self._write_mozconfig_json()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/www-client/firefox-100.0.1/work/firefox-100.0.1/python/mozbuild/mozbuild/controller/building.py", line 1767, in _write_mozconfig_json
    with FileAvoidWrite(mozconfig_json) as fh:
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/www-client/firefox-100.0.1/work/firefox-100.0.1/python/mozbuild/mozbuild/util.py", line 353, in __exit__
    self.close()
    ^^^^^^^^^^^^
  File "/var/tmp/portage/www-client/firefox-100.0.1/work/firefox-100.0.1/python/mozbuild/mozbuild/util.py", line 277, in close
    existing = _open(self.name, self.mode)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/www-client/firefox-100.0.1/work/firefox-100.0.1/python/mozbuild/mozbuild/util.py", line 63, in _open
    return io.open(path, mode, encoding="utf-8", newline="\n")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 * ERROR: www-client/firefox-100.0.1::gentoo failed (configure phase):

Expected results:

I'm flagging this bug early, since python-3.11 no longer accepts 'U'.
See
https://docs.python.org/3.11/whatsnew/3.11.html#changes-in-the-python-api
open(), io.open(), codecs.open() and fileinput.FileInput no longer accept 'U' (“universal newline”) in the file mode.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Mach Core' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[sevenmentor]

(spam)

Comment 3

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1769631 - Remove 'U' from 'mode' parameters for various 'open' calls to ensure Python3.11 compatibility r?#build

The 'U' flag represents "universal newline". It has been deprecated
since Python3.3. Since then "universal newline" is the default when a
file is opened in text mode (not bytes). In Python3.11 using the 'U'
flag throws errors. There should be no harm in removing 'U' from 'open'
everywhere it is used, and doing allows the use of Python3.11.

For more reading see: https://docs.python.org/3.11/whatsnew/3.11.html#changes-in-the-python-api

Comment 4

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

I used ["][rwx][abt+]{0,4}[U]["] as my regex to find all the U's on searchfox. Hopefully that's all of them.

Comment 5

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

I should have used ['"][rwx][abt+]{0,4}[U]['"], but luckily all the single quote usages are in third_party or web_platform, which are both outside the scope of what we can modify.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f54162b2c1f2
Remove 'U' from 'mode' parameters for various 'open' calls to ensure Python3.11 compatibility r=firefox-build-system-reviewers,glandium

Comment 7

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/f54162b2c1f2

Comment 8

[Joonas Niilola]

Attachment: firefox-101.0:20220603-062929.log

Hey,

thanks for the swift work on this! I tried applying the patch on top of 101.0 release. While I understand many things could've changed when comparing to 103, just wanted to point out 101.0 does not build with python-3.11. The full build.log is attached if you can see something obvious from it. To me it looks like it could be related to bundled python libs? That's just a guess, but wanted to point out that maybe they could be updated for 103 release, when not relying on pip install to install python dependencies.

Comment 9

[Mike Hommey [:glandium]]

Comment on attachment 9278881 [details]
Bug 1769631 - Remove 'U' from 'mode' parameters for various 'open' calls to ensure Python3.11 compatibility r?#build

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Build failure with python 3.11
• User impact if declined: See above
• Fix Landed on Version: 103
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Practically speaking, it's a no-op change.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9278881 [details]
Bug 1769631 - Remove 'U' from 'mode' parameters for various 'open' calls to ensure Python3.11 compatibility r?#build

Approved for 102.8esr.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/1345ee1bfd75