1769735 - Perma [tier 2] ModuleNotFoundError: No module named 'packaging'

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=378326745&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/A29aLg59TLevTo2oFfScRA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-05-17T10:34:18.150Z] 10:34:18     INFO -  TEST-PASS | tests/jit-test/jit-test/tests/basic/bug641563.js | Success (code 3, args "--baseline-eager") [0.2 s]
[task 2022-05-17T10:34:18.213Z] 10:34:18     INFO -  TEST-PASS | tests/jit-test/jit-test/tests/basic/bug641563.js | Success (code 3, args "--no-blinterp --no-baseline --no-ion --more-compartments") [0.2 s]
[task 2022-05-17T10:34:18.217Z] 10:34:18     INFO -  TEST-PASS | tests/jit-test/jit-test/tests/basic/bug641563.js | Success (code 3, args "--blinterp-eager") [0.2 s]
[task 2022-05-17T10:34:18.217Z] 10:34:18     INFO -  PASSED ALL
[task 2022-05-17T10:34:18.217Z] 10:34:18     INFO -  Result summary:
[task 2022-05-17T10:34:18.217Z] 10:34:18     INFO -  Passed: 7470
[task 2022-05-17T10:34:18.217Z] 10:34:18     INFO -  Failed: 0
[task 2022-05-17T10:34:18.238Z] 10:34:18     INFO - Return code: 0
[task 2022-05-17T10:34:18.238Z] 10:34:18     INFO - TinderboxPrint: jittest-jittest<br/>7470/0
[task 2022-05-17T10:34:18.238Z] 10:34:18     INFO - # TBPL SUCCESS #
[task 2022-05-17T10:34:18.238Z] 10:34:18     INFO - The jittest suite: jittest ran with return status: SUCCESS
[task 2022-05-17T10:34:18.238Z] 10:34:18     INFO - Running post-action listener: _package_coverage_data
[task 2022-05-17T10:34:18.250Z] 10:34:18    ERROR - Exception during post-action for run-tests: Traceback (most recent call last):
[task 2022-05-17T10:34:18.250Z] 10:34:18    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 2304, in run_action
[task 2022-05-17T10:34:18.250Z] 10:34:18    ERROR -     method(action, success=success and self.return_code == 0)
[task 2022-05-17T10:34:18.250Z] 10:34:18    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/codecoverage.py", line 510, in _package_coverage_data
[task 2022-05-17T10:34:18.250Z] 10:34:18    ERROR -     self.gcov_dir, self.jsvm_dir
[task 2022-05-17T10:34:18.250Z] 10:34:18    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/codecoverage.py", line 306, in parse_coverage_artifacts
[task 2022-05-17T10:34:18.250Z] 10:34:18    ERROR -     from codecoverage.lcov_rewriter import LcovFileRewriter
[task 2022-05-17T10:34:18.250Z] 10:34:18    ERROR -   File "/builds/worker/workspace/build/tests/mozbuild/codecoverage/lcov_rewriter.py", line 18, in <module>
[task 2022-05-17T10:34:18.251Z] 10:34:18    ERROR -     from mozpack.chrome.manifest import parse_manifest
[task 2022-05-17T10:34:18.251Z] 10:34:18    ERROR -   File "/builds/worker/workspace/build/tests/mozpack/chrome/manifest.py", line 12, in <module>
[task 2022-05-17T10:34:18.251Z] 10:34:18    ERROR -     from mozpack.chrome.flags import Flags
[task 2022-05-17T10:34:18.251Z] 10:34:18    ERROR -   File "/builds/worker/workspace/build/tests/mozpack/chrome/flags.py", line 9, in <module>
[task 2022-05-17T10:34:18.251Z] 10:34:18    ERROR -     from packaging.version import Version
[task 2022-05-17T10:34:18.251Z] 10:34:18    ERROR - ModuleNotFoundError: No module named 'packaging'
[task 2022-05-17T10:34:18.251Z] 10:34:18     INFO - Running post-action listener: _resource_record_post_action
[task 2022-05-17T10:34:18.251Z] 10:34:18     INFO - Running post-action listener: process_java_coverage_data
[task 2022-05-17T10:34:18.251Z] 10:34:18     INFO - [mozharness: 2022-05-17 10:34:18.251291Z] Finished run-tests step (success)
[task 2022-05-17T10:34:18.251Z] 10:34:18    FATAL - Aborting due to failure in post-action listener.
[task 2022-05-17T10:34:18.251Z] 10:34:18    FATAL - Running post_fatal callback...
[task 2022-05-17T10:34:18.251Z] 10:34:18    FATAL - Exiting -1
[task 2022-05-17T10:34:18.251Z] 10:34:18     INFO - Running post-run listener: _resource_record_post_run
[task 2022-05-17T10:34:18.315Z] 10:34:18     INFO - Validating Perfherder data against /builds/worker/workspace/mozharness/external_tools/performance-artifact-schema.json
[task 2022-05-17T10:34:18.318Z] 10:34:18     INFO - PERFHERDER_DATA: {"framework": {"name": "job_resource_usage"}, "suites": [{"name": "jittest.jittest.1.overall", "extraOptions": ["taskcluster-m5d.xlarge"], "subtests": [{"name": "cpu_percent", "value": 91.28448741007196}, {"name": "io_write_bytes", "value": 3758243840}, {"name": "io.read_bytes", "value": 9396224}, {"name": "io_write_time", "value": 357476}, {"name": "io_read_time", "value": 180}]}, {"name": "jittest.jittest.1.start-pulseaudio", "subtests": [{"name": "time", "value": 0.016794204711914062}]}, {"name": "jittest.jittest.1.install", "subtests": [{"name": "time", "value": 57.036961793899536}, {"name": "cpu_percent", "value": 25.259090909090908}]}, {"name": "jittest.jittest.1.stage-files", "subtests": [{"name": "time", "value": 0.00032210350036621094}]}, {"name": "jittest.jittest.1.run-tests", "subtests": [{"name": "time", "value": 500.00229716300964}, {"name": "cpu_percent", "value": 98.82625250501005}]}]}
[task 2022-05-17T10:34:18.318Z] 10:34:18     INFO - Total resource usage - Wall time: 557s; CPU: 99%; Read bytes: 9396224; Write bytes: 3758243840; Read time: 180; Write time: 357476
[task 2022-05-17T10:34:18.318Z] 10:34:18     INFO - TinderboxPrint: CPU usage<br/>98.8%
[task 2022-05-17T10:34:18.318Z] 10:34:18     INFO - TinderboxPrint: I/O read bytes / time<br/>9,396,224 / 180
[task 2022-05-17T10:34:18.318Z] 10:34:18     INFO - TinderboxPrint: I/O write bytes / time<br/>3,758,243,840 / 357,476
[task 2022-05-17T10:34:18.318Z] 10:34:18     INFO - TinderboxPrint: CPU idle<br/>186.8 (8.4%)
[task 2022-05-17T10:34:18.319Z] 10:34:18     INFO - TinderboxPrint: CPU system<br/>220.7 (9.9%)
[task 2022-05-17T10:34:18.319Z] 10:34:18     INFO - TinderboxPrint: CPU user<br/>1,807.5 (81.3%)
[task 2022-05-17T10:34:18.319Z] 10:34:18     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2022-05-17T10:34:18.319Z] 10:34:18     INFO - start-pulseaudio - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2022-05-17T10:34:18.319Z] 10:34:18     INFO - install - Wall time: 57s; CPU: 25%; Read bytes: 8192; Write bytes: 2232184832; Read time: 72; Write time: 273288
[task 2022-05-17T10:34:18.320Z] 10:34:18     INFO - stage-files - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2022-05-17T10:34:18.323Z] 10:34:18     INFO - run-tests - Wall time: 500s; CPU: 99%; Read bytes: 9388032; Write bytes: 1526059008; Read time: 108; Write time: 84188
[task 2022-05-17T10:34:18.429Z] cleanup
[task 2022-05-17T10:34:18.429Z] + cleanup
[task 2022-05-17T10:34:18.430Z] + local rv=255
[task 2022-05-17T10:34:18.430Z] + [[ -s /builds/worker/.xsession-errors ]]
[task 2022-05-17T10:34:18.430Z] + cp /builds/worker/.xsession-errors /builds/worker/artifacts/public/xsession-errors.log
[task 2022-05-17T10:34:18.431Z] + '[' ']'
[task 2022-05-17T10:34:18.431Z] + true
[task 2022-05-17T10:34:18.431Z] + cleanup_xvfb
[task 2022-05-17T10:34:18.432Z] ++ pidof Xvfb
[task 2022-05-17T10:34:18.435Z] + local xvfb_pid=53
[task 2022-05-17T10:34:18.435Z] + local vnc=false
[task 2022-05-17T10:34:18.435Z] + local interactive=false
[task 2022-05-17T10:34:18.435Z] + '[' -n 53 ']'
[task 2022-05-17T10:34:18.435Z] + [[ false == false ]]
[task 2022-05-17T10:34:18.435Z] + [[ false == false ]]
[task 2022-05-17T10:34:18.435Z] + kill 53
[task 2022-05-17T10:34:18.436Z] + screen -XS xvfb quit
[task 2022-05-17T10:34:18.445Z] + exit 255
[taskcluster 2022-05-17 10:34:18.901Z] === Task Finished ===
[taskcluster 2022-05-17 10:34:21.144Z] Unsuccessful task run with exit code: 255 completed in 682.658 seconds

Comment 1

[Norisz Fay [:noriszfay]]

Hey Ryan, this started perma failing as tier 2 on central only so far on this merge and it seems to be caused by Bug 1766085 . Could you take a look please? Thank you!

Comment 2

[Ryan VanderMeulen [:RyanVM]]

I'm confused - packaging is vendored in-tree. Andrew, do you know why our ccov jobs wouldn't be using it?

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1766085

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1769735 - Revert packaging.version changes to flags.py due to CCov test bustage.

Comment 5

[Intermittent Failures Robot]

643 failures in 667 pushes (0.964 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 643

Platform and build breakdown:

• linux1804-64-ccov-qr: 286
  • opt: 286
• windows10-64-2004-ccov-qr: 357
  • opt: 357

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1769735&startday=2022-05-17&endday=2022-05-17&tree=all

Comment 6

[ISHIKAWA, Chiaki]

FYI, I see the same warning when I build TB under Debian GNU/Linux.
I have frozen my local M-C/C-C tree since last Friday to track down bugs caused by my local patches in my local testing.
The message is the same.

/NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/build/bin/python -m mozbuild.action.check_binary --target libxul.so
/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozbuild/mozbuild/action/check_binary.py:13: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.version import StrictVersion as Version
chmod +x libxul.so

I also have

ishikawa@ip030:/home/ishikawa$ python --version
Python 3.10.4
ishikawa@ip030:/home/ishikawa$ 

Comment 7

[Andrew Halberstadt [:ahal]]

The test tasks run from a "test bundle" rather than a source checkout, so likely it isn't being copied in there. You can inspect the contents by looking for the target.common.tests.tar.gz artifact on the build (there's also suite specific packages).

To get additional packages copied into this artifact, you can add an entry like this:
https://searchfox.org/mozilla-central/rev/8fe6930c0832009b3162bebee7d4ede1a4c8c9a8/python/mozbuild/mozbuild/action/test_archive.py#174

Add it to the common package if it's needed by all harnesses, or add it to a suite specific package if only needed by a certain test suite.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1769735 - Add packaging to the mozharness archive.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

(In reply to Andrew Halberstadt [:ahal] from comment #7)

Add it to the common package if it's needed by all harnesses, or add it to a suite specific package if only needed by a certain test suite.

Adding it to the mozharness package works, thanks!
https://treeherder.mozilla.org/jobs?repo=try&revision=455f33748b753d8baa500031ce44fe199f513836

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/51b2a37757b1
Add packaging to the mozharness archive. r=aki

Comment 11

[Intermittent Failures Robot]

641 failures in 585 pushes (1.096 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 641

Platform and build breakdown:

• linux1804-64-ccov-qr: 282
  • opt: 282
• windows10-64-2004-ccov-qr: 359
  • opt: 359

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1769735&startday=2022-05-18&endday=2022-05-18&tree=all

Comment 12

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/51b2a37757b1

Comment 13

[Intermittent Failures Robot]

1286 failures in 2694 pushes (0.477 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 1286

Platform and build breakdown:

• linux1804-64-ccov-qr: 568
  • opt: 568
• windows10-64-2004-ccov-qr: 718
  • opt: 718

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1769735&startday=2022-05-16&endday=2022-05-22&tree=all