1925198 - Nightly fails to build with python-3.13: ModuleNotFoundError: No module named 'telnetlib'

Status: RESOLVED FIXED

(Testing :: Mozbase, defect, P2)

Description

[matthias koplenig [:metasieben]]

Similarly to pipes, telnetlib has been removed in Python 3.13
https://docs.python.org/3.12/library/telnetlib.html

Building Nightly with PGO enabled and Python 3.13 is currently broken.

https://searchfox.org/mozilla-central/rev/aecbd5cdd28a09e11872bc829d9e6e4b943e6e49/testing/mozbase/mozrunner/mozrunner/devices/emulator.py#11

Comment 1

[Joonas Niilola]

I was going to open a bug about this as well. Python-3.13 seems to work otherwise, but not when pgo is enabled via MOZ_PGO=1.

I get this error:

21:02.91 gmake: Leaving directory '/var/tmp/portage/www-client/firefox-131.0.3/work/firefox_build/instrumented'
Traceback (most recent call last):
  File "/var/tmp/portage/www-client/firefox-131.0.3/work/firefox-131.0.3/build/pgo/profileserver.py", line 19, in <module>
    from mozrunner import CLI, FirefoxRunner
  File "/var/tmp/portage/www-client/firefox-131.0.3/work/firefox-131.0.3/testing/mozbase/mozrunner/mozrunner/__init__.py", line 6, in <module>
    import mozrunner.base
  File "/var/tmp/portage/www-client/firefox-131.0.3/work/firefox-131.0.3/testing/mozbase/mozrunner/mozrunner/base/__init__.py", line 6, in <module>
    from .browser import BlinkRuntimeRunner, GeckoRuntimeRunner
  File "/var/tmp/portage/www-client/firefox-131.0.3/work/firefox-131.0.3/testing/mozbase/mozrunner/mozrunner/base/browser.py", line 11, in <module>
    from ..application import DefaultContext, FirefoxContext
  File "/var/tmp/portage/www-client/firefox-131.0.3/work/firefox-131.0.3/testing/mozbase/mozrunner/mozrunner/application.py", line 11, in <module>
    from mozdevice import ADBDeviceFactory
  File "/var/tmp/portage/www-client/firefox-131.0.3/work/firefox-131.0.3/testing/mozbase/mozdevice/mozdevice/__init__.py", line 157, in <module>
    from .adb import (
    ...<8 lines>...
    )
  File "/var/tmp/portage/www-client/firefox-131.0.3/work/firefox-131.0.3/testing/mozbase/mozdevice/mozdevice/adb.py", line 7, in <module>
    import pipes
ModuleNotFoundError: No module named 'pipes'

And yeah, import pipes is still here:
https://hg.mozilla.org/mozilla-central/file/tip/testing/mozbase/mozdevice/mozdevice/adb.py

Comment 2

[Lukasz Gniadzik [:lgniadzik]]

(In reply to Joonas Niilola from comment #1)

And yeah, import pipes is still here:
https://hg.mozilla.org/mozilla-central/file/tip/testing/mozbase/mozdevice/mozdevice/adb.py

In this script we only use quote method from pipes module so we may replace that usage with it's equivalent from mozbuild.shellutil (as it has been done in other places)

from mozbuild.shellutil import quote as shell_quote

class ADBProcess:
    @staticmethod
    def _quote(arg):
        """Utility function to return quoted version of command argument."""
        return shell_quote(arg)

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jmaher, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

thanks, for all the info, this seems actionable!

Comment 5

[Rob Wu [:robwu]]

I fixed pipes issues in bug 1926140, but the missing telnetlib issue still needs to be resolved before commands like ./mach android-emulator can work again.

It looks like telnetlib is primarily used to connect to the emulator (source), e.g.:

• verifying that the emulator is connected
• response = self.emulator._run_telnet("power display")
• self.emulator._run_telnet("geo fix %0.5f %0.5f" % (self.lon, self.lat))
• response = self.emulator._run_telnet("sensor get orientation")

I don't see any test actually modifying the geolocation or battery level, but there is a marionette test that checks screen orientation.

I wonder whether we could get rid of the telnet dependency and use adb directly to (1) check whether the emulator is ready and (2) change screen orientation.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I like the idea of using adb instead of telnet. The new telnetlib is async and would be incongruous with our existing scripts, so not as straightforward as a few lines of code changed. Also to test, mozrunner is baked into hostutils, so testing this requires bumping the version number of mozrunner in your patch and ensuring that our scripts depend on the new version number.

Comment 7

[Rob Wu [:robwu]]

Another approach is to see if there is a minimum set of telnet functionality (from https://github.com/python/cpython/blob/3.11/Lib/telnetlib.py or from scratch) that we can fork into a new file. The advantage of it is that it would be less risky in terms of causing unexpected breakage, because we would not be changing the behavior of Android.

And I noticed that even with the fix from bug 1926140, that I am unable to run tests on desktop, despite it being independent of the Android emulator. This is because the telnetlib import is at the top level and unconditional. As a stop-gap measure to unblock testing on desktop, I am going to submit a patch that moves the top level import to within the function, so that we can still run xpcshell tests on desktop.

Comment 8

[Rob Wu [:robwu]]

Relevant stack for comment 7 when running a xpcshell test through mach test:

  File "/path/to/testing/xpcshell/mach_commands.py", line 50, in _run_xpcshell_harness
    import runxpcshelltests
  File "/path/to/testing/xpcshell/runxpcshelltests.py", line 97, in <module>
    from mozrunner.utils import get_stack_fixer_function
  File "/path/to/testing/mozbase/mozrunner/mozrunner/__init__.py", line 6, in <module>
    import mozrunner.base
  File "/path/to/testing/mozbase/mozrunner/mozrunner/base/__init__.py", line 7, in <module>
    from .device import DeviceRunner, FennecRunner
  File "/path/to/testing/mozbase/mozrunner/mozrunner/base/device.py", line 16, in <module>
    from ..devices import BaseEmulator
  File "/path/to/testing/mozbase/mozrunner/mozrunner/devices/__init__.py", line 8, in <module>
    from .emulator import BaseEmulator, EmulatorAVD
  File "/path/to/testing/mozbase/mozrunner/mozrunner/devices/emulator.py", line 11, in <module>
    from telnetlib import Telnet

Comment 9

[Rob Wu [:robwu]]

Attachment: Bug 1925198 - Move telnetlib import to BaseEmulator

emulator.py currently has an unconditional telnetlib import, but the
file can also be imported when the functionality is not needed.

Because telnetlib was removed from Python 3.13, this unconditional
import breaks use cases that do not even care about telnet. As a stopgap
measure until a better fix is available, move the import to the function
that relies on telnet.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/50d41051085b
Move telnetlib import to BaseEmulator r=jmaher

Comment 11

[Rob Wu [:robwu]]

Attachment: Bug 1925198 - Vendor telnetlib.py

telnetlib.py was removed from Python 3.13, but we still depend on its
functionality. This patch imports the original standard library from
https://github.com/python/cpython/blob/3.12/Lib/telnetlib.py

The only changes are:

• Prepend comments at start to attribute source.
• Prepend fmt/noqa/ruff to suppress linting by black/ruff, to avoid
  non-critical modifications.
• Remove warnings._deprecated(__name__, remove=(3, 13)) call to
  avoid a deprecation warning when loading the module.

Comment 12

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/50d41051085b

Comment 13

[Rob Wu [:robwu]]

The first patch solves the reported issue (unbreaking the builds on desktop), but there is still an issue with Android builds/tests. I'll move the second patch to a new bug.

Comment 14

[Phabricator Automation]

Comment on attachment 9435489 [details]
Bug 1925198 - Vendor telnetlib.py

Revision D228052 was moved to bug 1929535. Setting attachment 9435489 [details] to obsolete.