Closed Bug 1922075 Opened 1 year ago Closed 1 year ago

mozinstall failing on 24.04 due to six.moves package

Tracking

(firefox133 fixed)

Status:

RESOLVED FIXED

Tracking Flags:

Tracking

Status

firefox133

---

fixed

People

(Reporter: jmoss, Assigned: ahal)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

Bug 1922075 - Remove uses of 'six.moves' from testing/mozbase, r?#mozbase-reviewers 1 year ago Andrew Halberstadt [:ahal] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1922075 - mozharness: install six 1.16.0 instead of 1.13.0 1 year ago Julien Cristau [:jcristau] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1922075 - install newer requests python package for compatibility with python 3.12 1 year ago Julien Cristau [:jcristau] 48 bytes, text/x-phabricator-request		Details \| Review

:jmoss

Reporter

Description

•

1 year ago

When running mozinstall on linux 24.04 wayland image, there's an error with six.modules.

[task 2024-10-01T14:11:14.675Z] 14:11:14    ERROR - Errors received:
[task 2024-10-01T14:11:14.675Z] 14:11:14    ERROR -  Traceback (most recent call last):
[task 2024-10-01T14:11:14.675Z] 14:11:14    ERROR -    File "/home/task_172779168574566/build/venv/bin/mozinstall", line 5, in <module>
[task 2024-10-01T14:11:14.675Z] 14:11:14    ERROR -      from mozinstall import install_cli
[task 2024-10-01T14:11:14.675Z] 14:11:14    ERROR -    File "/home/task_172779168574566/build/venv/lib/python3.12/site-packages/mozinstall/__init__.py", line 6, in <module>
[task 2024-10-01T14:11:14.675Z] 14:11:14    ERROR -      from .mozinstall import *
[task 2024-10-01T14:11:14.675Z] 14:11:14    ERROR -    File "/home/task_172779168574566/build/venv/lib/python3.12/site-packages/mozinstall/mozinstall.py", line 16, in <module>
[task 2024-10-01T14:11:14.676Z] 14:11:14    ERROR -      import mozfile
[task 2024-10-01T14:11:14.676Z] 14:11:14    ERROR -    File "/home/task_172779168574566/build/venv/lib/python3.12/site-packages/mozfile/__init__.py", line 6, in <module>
[task 2024-10-01T14:11:14.676Z] 14:11:14    ERROR -      from .mozfile import *
[task 2024-10-01T14:11:14.676Z] 14:11:14    ERROR -    File "/home/task_172779168574566/build/venv/lib/python3.12/site-packages/mozfile/mozfile.py", line 17, in <module>
[task 2024-10-01T14:11:14.676Z] 14:11:14    ERROR -      from six.moves import urllib
[task 2024-10-01T14:11:14.676Z] 14:11:14    ERROR -  ModuleNotFoundError: No module named 'six.moves'

:jmoss

Reporter

Updated

•

1 year ago

Component: RelOps: Posix OS → General

Product: Infrastructure & Operations → Release Engineering

QA Contact: jlorenzo

:jmoss

Reporter

Updated

•

1 year ago

Flags: needinfo?(ahal)

:jmoss

Reporter

Comment 1

•

1 year ago

Livelog: https://firefox-ci-tc.services.mozilla.com/tasks/GrWST-sUSCi11yJ83VnPOw/runs/2/logs/public/logs/live.log

Andrew Halberstadt [:ahal]

Assignee

Comment 2

•

1 year ago

I'll see if I can remove our dependency on six altogether in bug 1922081.

Depends on: 1922081

Flags: needinfo?(ahal)

Andrew Halberstadt [:ahal]

Assignee

Comment 3

•

1 year ago

Actually I'll just remove six.moves in this bug for now as that will have a smaller chance of causing a regression. Then we can tackle the rest of it in the other bug.

No longer depends on: 1922081

Andrew Halberstadt [:ahal]

Assignee

Comment 4

•

1 year ago

Attached file Bug 1922075 - Remove uses of 'six.moves' from testing/mozbase, r?#mozbase-reviewers — Details

Phabricator Automation

Updated

•

1 year ago

Assignee: nobody → ahal

Status: NEW → ASSIGNED

Pulsebot

Comment 5

•

1 year ago

Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/6d1113bb77a8 Remove uses of 'six.moves' from testing/mozbase, r=mozbase-reviewers,jmaher

Iulian Moraru

Comment 6

•

1 year ago

Backed out for causing android Gtest failures.

[task 2024-10-01T23:56:24.830Z] 23:56:24     INFO - >> Install app: Attempt #1 of 5
[task 2024-10-01T23:56:36.381Z] 23:56:36     INFO - Finished installing apps for emulator-5554
[task 2024-10-01T23:56:36.381Z] 23:56:36     INFO - Running post-action listener: _resource_record_post_action
[task 2024-10-01T23:56:36.381Z] 23:56:36     INFO - [mozharness: 2024-10-01 23:56:36.381293Z] Finished install step (success)
[task 2024-10-01T23:56:36.381Z] 23:56:36     INFO - [mozharness: 2024-10-01 23:56:36.381340Z] Running run-tests step.
[task 2024-10-01T23:56:36.381Z] 23:56:36     INFO - Running pre-action listener: _resource_record_pre_action
[task 2024-10-01T23:56:36.381Z] 23:56:36     INFO - Running pre-action listener: _set_gcov_prefix
[task 2024-10-01T23:56:36.381Z] 23:56:36     INFO - Running pre-action listener: timed_screenshots
[task 2024-10-01T23:56:36.381Z] 23:56:36     INFO - Running main action method: run_tests
[task 2024-10-01T23:56:36.382Z] 23:56:36     INFO - Running the command /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/gtest/remotegtests.py --symbols-path=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EUCjJCCjTXWw0-HdKEp-EQ/artifacts/public/build/target.crashreporter-symbols.zip --tests-path=/builds/worker/workspace/build/tests/gtest --libxul=/builds/worker/workspace/build/tests/gtest/gtest_bin/gtest/libxul.so --package=org.mozilla.geckoview.test_runner --deviceSerial=emulator-5554
[task 2024-10-01T23:56:36.382Z] 23:56:36     INFO - ##### gtest log begins
[task 2024-10-01T23:56:36.382Z] 23:56:36     INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/gtest/remotegtests.py', '--symbols-path=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EUCjJCCjTXWw0-HdKEp-EQ/artifacts/public/build/target.crashreporter-symbols.zip', '--tests-path=/builds/worker/workspace/build/tests/gtest', '--libxul=/builds/worker/workspace/build/tests/gtest/gtest_bin/gtest/libxul.so', '--package=org.mozilla.geckoview.test_runner', '--deviceSerial=emulator-5554'] in /builds/worker/workspace/build/tests/gtest
[task 2024-10-01T23:56:36.382Z] 23:56:36     INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/gtest/remotegtests.py --symbols-path=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EUCjJCCjTXWw0-HdKEp-EQ/artifacts/public/build/target.crashreporter-symbols.zip --tests-path=/builds/worker/workspace/build/tests/gtest --libxul=/builds/worker/workspace/build/tests/gtest/gtest_bin/gtest/libxul.so --package=org.mozilla.geckoview.test_runner --deviceSerial=emulator-5554
[task 2024-10-01T23:56:36.382Z] 23:56:36     INFO - Using env: (same as previous command)
[task 2024-10-01T23:56:36.499Z] 23:56:36     INFO -  gtest ERROR | module 'urllib' has no attribute 'parse'
[task 2024-10-01T23:56:36.499Z] 23:56:36     INFO -  Traceback (most recent call last):
[task 2024-10-01T23:56:36.499Z] 23:56:36     INFO -    File "/builds/worker/workspace/build/tests/gtest/remotegtests.py", line 460, in main
[task 2024-10-01T23:56:36.499Z] 23:56:36     INFO -      result = tester.run_gtest(
[task 2024-10-01T23:56:36.499Z] 23:56:36     INFO -    File "/builds/worker/workspace/build/tests/gtest/remotegtests.py", line 81, in run_gtest
[task 2024-10-01T23:56:36.499Z] 23:56:36     INFO -      update_mozinfo()
[task 2024-10-01T23:56:36.499Z] 23:56:36     INFO -    File "/builds/worker/workspace/build/tests/gtest/remotegtests.py", line 442, in update_mozinfo
[task 2024-10-01T23:56:36.499Z] 23:56:36     INFO -      mozinfo.find_and_update_from_json(*dirs)
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozinfo/mozinfo.py", line 284, in find_and_update_from_json
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -      update(json_path)
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozinfo/mozinfo.py", line 228, in update
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -      f = mozfile.load(new_info)
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozfile/mozfile.py", line 632, in load
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -      if not is_url(resource):
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -    File "/builds/worker/workspace/build/venv/lib/python3.8/site-packages/mozfile/mozfile.py", line 614, in is_url
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -      parsed = urllib.parse.urlparse(thing)
[task 2024-10-01T23:56:36.500Z] 23:56:36     INFO -  AttributeError: module 'urllib' has no attribute 'parse'
[task 2024-10-01T23:56:36.507Z] 23:56:36     INFO - Return code: 1
[task 2024-10-01T23:56:36.507Z] 23:56:36    ERROR - No tests run or test summary not found
[task 2024-10-01T23:56:36.508Z] 23:56:36     INFO - TinderboxPrint: gtest<br/><em class="testfail">T-FAIL</em>
[task 2024-10-01T23:56:36.508Z] 23:56:36     INFO - ##### gtest log ends
[task 2024-10-01T23:56:36.508Z] 23:56:36  WARNING - setting return code to 1
[task 2024-10-01T23:56:36.508Z] 23:56:36     INFO - The gtest suite: gtest ran with return status: WARNING
[task 2024-10-01T23:56:36.508Z] 23:56:36     INFO - Running post-action listener: _package_coverage_data
[task 2024-10-01T23:56:36.508Z] 23:56:36     INFO - Running post-action listener: _resource_record_post_action
[task 2024-10-01T23:56:36.508Z] 23:56:36     INFO - Running post-action listener: process_java_coverage_data
[task 2024-10-01T23:56:36.508Z] 23:56:36     INFO - Running post-action listener: stop_device
[task 2024-10-01T23:56:36.774Z] 23:56:36     INFO - Killing logcat pid 1380.
[task 2024-10-01T23:56:36.774Z] 23:56:36     INFO - Killing every process called qemu-system-x86_64
[task 2024-10-01T23:56:36.779Z] 23:56:36     INFO - [mozharness: 2024-10-01 23:56:36.779206Z] Finished run-tests step (success)
[task 2024-10-01T23:56:36.779Z] 23:56:36     INFO - Running post-run listener: _resource_record_post_run
[task 2024-10-01T23:56:36.827Z] 23:56:36     INFO - Total resource usage - Wall time: 43s; CPU: 26%; Read bytes: 0; Write bytes: 301494272; Read time: 0; Write time: 277524
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - TinderboxPrint: CPU usage<br/>26.5%
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - TinderboxPrint: I/O read bytes / time<br/>0 / 0
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - TinderboxPrint: I/O write bytes / time<br/>301,494,272 / 277,524
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - TinderboxPrint: CPU guest<br/>48.7 (12.6%)
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - TinderboxPrint: CPU idle<br/>244.9 (63.6%)
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - TinderboxPrint: CPU system<br/>27.5 (7.1%)
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - TinderboxPrint: CPU user<br/>62.1 (16.1%)
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2024-10-01T23:56:36.828Z] 23:56:36     INFO - start-emulator - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-10-01T23:56:36.830Z] 23:56:36     INFO - verify-device - Wall time: 31s; CPU: 24%; Read bytes: 0; Write bytes: 9461760; Read time: 0; Write time: 6516
[task 2024-10-01T23:56:36.831Z] 23:56:36     INFO - install - Wall time: 12s; CPU: 33%; Read bytes: 0; Write bytes: 292016128; Read time: 0; Write time: 270964
[task 2024-10-01T23:56:36.832Z] 23:56:36     INFO - run-tests - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-10-01T23:56:36.922Z] 23:56:36  WARNING - returning nonzero exit status 1
[task 2024-10-01T23:56:36.950Z] cleanup
[task 2024-10-01T23:56:36.950Z] + cleanup
[task 2024-10-01T23:56:36.950Z] + local rv=1
[task 2024-10-01T23:56:36.950Z] + [[ -s /builds/worker/.xsession-errors ]]
[task 2024-10-01T23:56:36.950Z] + cp /builds/worker/.xsession-errors /builds/worker/artifacts/public/xsession-errors.log
[task 2024-10-01T23:56:36.952Z] + '[' ']'
[task 2024-10-01T23:56:36.952Z] + true
[task 2024-10-01T23:56:36.952Z] + cleanup_xvfb
[task 2024-10-01T23:56:36.952Z] ++ pidof Xvfb
[task 2024-10-01T23:56:36.954Z] + local xvfb_pid=58
[task 2024-10-01T23:56:36.955Z] + local vnc=false
[task 2024-10-01T23:56:36.955Z] + local interactive=false
[task 2024-10-01T23:56:36.955Z] + '[' -n 58 ']'
[task 2024-10-01T23:56:36.955Z] + [[ false == false ]]
[task 2024-10-01T23:56:36.955Z] + [[ false == false ]]
[task 2024-10-01T23:56:36.955Z] + kill 58
[task 2024-10-01T23:56:36.955Z] + screen -XS xvfb quit
[task 2024-10-01T23:56:36.963Z] + exit 1
[taskcluster 2024-10-01 23:56:37.633Z] === Task Finished ===
[taskcluster 2024-10-01 23:56:39.250Z] Unsuccessful task run with exit code: 1 completed in 174.89 seconds

Flags: needinfo?(ahal)

Andrew Halberstadt [:ahal]

Assignee

Comment 7

•

1 year ago

Oh right, I forgot that urllib lazy loads the parse and request submodules. So you need to do either import urllib.parse or from urllib.parse import ....

Flags: needinfo?(ahal)

bhearsum@mozilla.com (:bhearsum)

Updated

•

1 year ago

Severity: -- → S3

Priority: -- → P2

Pulsebot

Comment 8

•

1 year ago

Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/9e6a490c9de6 Remove uses of 'six.moves' from testing/mozbase, r=mozbase-reviewers,jmaher

Cristian Tuns

Comment 9

•

1 year ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/9e6a490c9de6

Status: ASSIGNED → RESOLVED

Closed: 1 year ago

status-firefox133: --- → fixed

Resolution: --- → FIXED

Joel Maher ( :jmaher ) (UTC -8)

Comment 10

•

1 year ago

it appears we still have a reference to six.moves as part of import requests which then requests urllib3:
https://treeherder.mozilla.org/jobs?repo=try&revision=f6ba5ccbbf7d96bceafaf556966a6b1c7edfc2e7&selectedTaskRun=VuyjS1XlTrChiI5bu4b-yA.0

specifically when we import connectionpool:
https://searchfox.org/mozilla-central/source/third_party/python/pip/pip/_vendor/urllib3/connectionpool.py#38

We should consider upgrading urllib3 (and probably requests). this is specifically in the mozinstall module, but I assume is used in many modules. I am not sure when we determine to "vendor" a package in vs use something on pypi.

:ahal, do you have thoughts?

Flags: needinfo?(ahal)

:jmoss

Reporter

Updated

•

1 year ago

Status: RESOLVED → REOPENED

Resolution: FIXED → ---

BugBot [:suhaib / :marco/ :calixte]

Updated

•

1 year ago

status-firefox133: fixed → affected

Julien Cristau [:jcristau]

Updated

•

1 year ago

Depends on: 1924695

Joel Maher ( :jmaher ) (UTC -8)

Comment 11

•

1 year ago

we still have failures: https://treeherder.mozilla.org/logviewer?job_id=478320680&repo=try&lineNumber=1247. to figure out how to stop six 1.13.0 and urllib 1.25.7 and requests 2.22.0 from being used.

possibly:
https://searchfox.org/mozilla-central/source/testing/mozharness/configs/unittests/linux_unittest.py#37

Julien Cristau [:jcristau]

Comment 12

•

1 year ago

AFAICT requests 2.22.0 comes from https://searchfox.org/mozilla-central/rev/754074e05178e017ef6c3d8e30428ffa8f1b794d/testing/config/mozbase_requirements.txt#29; testing a fix on try...

Julien Cristau [:jcristau]

Comment 13

•

1 year ago

Attached file Bug 1922075 - mozharness: install six 1.16.0 instead of 1.13.0 — Details

The latter is incompatible with python 3.12.

Julien Cristau [:jcristau]

Comment 14

•

1 year ago

Attached file Bug 1922075 - install newer requests python package for compatibility with python 3.12 — Details

Julien Cristau [:jcristau]

Updated

•

1 year ago

Blocks: 1925029

Pulsebot

Comment 15

•

1 year ago

Pushed by jcristau@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/62b41093e0d5 mozharness: install six 1.16.0 instead of 1.13.0 r=jmaher https://hg.mozilla.org/integration/autoland/rev/e6fc752db13e install newer requests python package for compatibility with python 3.12 r=mozbase-reviewers,jmaher

Cosmin Sabou [:CosminS]

Comment 16

•

1 year ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/62b41093e0d5
https://hg.mozilla.org/mozilla-central/rev/e6fc752db13e

Status: REOPENED → RESOLVED

Closed: 1 year ago → 1 year ago

status-firefox133: affected → fixed

Resolution: --- → FIXED

Andrew Halberstadt [:ahal]

Assignee

Comment 17

•

1 year ago

Thanks Julien!

Flags: needinfo?(ahal)

You need to log in before you can comment on or make changes to this bug.