1875050 - Perma condprof ModuleNotFoundError: No module named 'yaml'

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=443606342&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/chiw34IyTY6qbDZ3LiZT8g/runs/0/artifacts/public/logs/live_backing.log

---

[taskcluster 2024-01-17T11:35:05.273Z] Worker Type (releng-hardware/gecko-t-osx-1015-r8) settings:
[taskcluster 2024-01-17T11:35:05.273Z]   {
[taskcluster 2024-01-17T11:35:05.273Z]     "arch": "x86_64",
[taskcluster 2024-01-17T11:35:05.273Z]     "config": {
[taskcluster 2024-01-17T11:35:05.273Z]       "deploymentId": ""
[taskcluster 2024-01-17T11:35:05.273Z]     },
[taskcluster 2024-01-17T11:35:05.273Z]     "disk_size": "233.47 GiB",
[taskcluster 2024-01-17T11:35:05.273Z]     "generic-worker": {
[taskcluster 2024-01-17T11:35:05.273Z]       "engine": "simple",
[taskcluster 2024-01-17T11:35:05.273Z]       "go-arch": "amd64",
[taskcluster 2024-01-17T11:35:05.273Z]       "go-os": "darwin",
[taskcluster 2024-01-17T11:35:05.273Z]       "go-version": "go1.21.1",
[taskcluster 2024-01-17T11:35:05.273Z]       "release": "https://github.com/taskcluster/taskcluster/releases/tag/v55.3.4",
[taskcluster 2024-01-17T11:35:05.273Z]       "revision": "a71f300fcbcbae92395d7dfea9cc6f3d38007477",
[taskcluster 2024-01-17T11:35:05.273Z]       "source": "https://github.com/taskcluster/taskcluster/commits/a71f300fcbcbae92395d7dfea9cc6f3d38007477",
[taskcluster 2024-01-17T11:35:05.273Z]       "version": "55.3.4"
[taskcluster 2024-01-17T11:35:05.273Z]     },
[taskcluster 2024-01-17T11:35:05.273Z]     "ip": "10.49.58.132",
[taskcluster 2024-01-17T11:35:05.273Z]     "machine-setup": {
[taskcluster 2024-01-17T11:35:05.273Z]       "config": "https://github.com/mozilla-platform-ops/ronin_puppet"
[taskcluster 2024-01-17T11:35:05.273Z]     },
[taskcluster 2024-01-17T11:35:05.273Z]     "memory": "16 GB",
[taskcluster 2024-01-17T11:35:05.273Z]     "model_identifier": "Macmini8,1",
[taskcluster 2024-01-17T11:35:05.273Z]     "processor_cores": "6",
[taskcluster 2024-01-17T11:35:05.273Z]     "processor_count": "1",
[taskcluster 2024-01-17T11:35:05.273Z]     "processor_name": "6-Core Intel Core i7",
[taskcluster 2024-01-17T11:35:05.273Z]     "processor_speed": "3.2 GHz",
[taskcluster 2024-01-17T11:35:05.273Z]     "system_version": "macOS 10.15.7 (19H2026)",
[taskcluster 2024-01-17T11:35:05.273Z]     "workerGroup": "mdc1",
[taskcluster 2024-01-17T11:35:05.273Z]     "workerId": "macmini-r8-30"
[taskcluster 2024-01-17T11:35:05.273Z]   }
[taskcluster 2024-01-17T11:35:05.274Z] Task ID: chiw34IyTY6qbDZ3LiZT8g
[taskcluster 2024-01-17T11:35:05.274Z] === Task Starting ===
[taskcluster 2024-01-17T11:35:07.606Z] Uploading redirect artifact public/logs/live.log to URL https://firefoxci-websocktunnel.services.mozilla.com/mdc1.macmini-r8-30.60099/log/WzaaTLmkQbGkAwe7cTAB1g with mime type "text/plain; charset=utf-8" and expiry 2024-01-17T14:50:07.546Z
[taskcluster:warn 2024-01-17T11:35:08.704Z] [mounts] No SHA256 specified in task mounts for urlcontent:http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fk1aSUVYS2GtTZa1M556AQ/artifacts/public/run-task - SHA256 from downloaded file /opt/worker/downloads/DI5RaqU8Riyz1w8goHQTow is c52fd4fedd061ea8b8e343e5b5aea0218f33d8a0258c34a7a3a3ba02c0d14a32.
[taskcluster 2024-01-17T11:35:08.704Z] [mounts] Creating directory /opt/worker/tasks/task_170549130489610 with permissions 0700
[taskcluster 2024-01-17T11:35:08.704Z] [mounts] Copying /opt/worker/downloads/DI5RaqU8Riyz1w8goHQTow to /opt/worker/tasks/task_170549130489610/run-task
[taskcluster:warn 2024-01-17T11:35:08.789Z] [mounts] No SHA256 specified in task mounts for urlcontent:http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fk1aSUVYS2GtTZa1M556AQ/artifacts/public/fetch-content - SHA256 from downloaded file /opt/worker/downloads/e9EtEVW2TsepLJb0c9QE_A is d1489e34d8e284a5e89322d8840baabe4bb3deb6c6a4f6cbead6c8b1c5997022.
[taskcluster 2024-01-17T11:35:08.789Z] [mounts] Creating directory /opt/worker/tasks/task_170549130489610 with permissions 0700
[taskcluster 2024-01-17T11:35:08.789Z] [mounts] Copying /opt/worker/downloads/e9EtEVW2TsepLJb0c9QE_A to /opt/worker/tasks/task_170549130489610/fetch-content
[taskcluster 2024-01-17T11:35:08.822Z] Executing command 0: chmod '+x' run-task
[taskcluster 2024-01-17T11:35:08.828Z]    Exit Code: 0
[taskcluster 2024-01-17T11:35:08.828Z]    User Time: 615µs
[taskcluster 2024-01-17T11:35:08.828Z]  Kernel Time: 1.246ms
[taskcluster 2024-01-17T11:35:08.828Z]    Wall Time: 5.193ms
[taskcluster 2024-01-17T11:35:08.828Z]       Result: SUCCEEDED
[taskcluster 2024-01-17T11:35:08.828Z] Executing command 1: /usr/local/bin/python3 run-task --task-cwd fetches/condprofile -- bash -cx 'python3 -m venv . && bin/python3 condprof/main.py  --force-new  --geckodriver ${MOZ_FETCHES_DIR}/geckodriver --firefox ${MOZ_FETCHES_DIR}/target.dmg --scenario settled ../../archive'
[setup 2024-01-17T11:35:10.983Z] run-task started in /opt/worker/tasks/task_170549130489610
[setup 2024-01-17T11:35:11.015Z] MOZ_FETCHES_DIR is /opt/worker/tasks/task_170549130489610/fetches
[setup 2024-01-17T11:35:11.015Z] MOZ_PYTHON_HOME is /opt/worker/tasks/task_170549130489610/fetches/python
[fetches 2024-01-17T11:35:11.015Z] fetching artifacts
[fetches 2024-01-17T11:35:11.015Z] executing ['/usr/local/bin/python3', '-u', 'fetch-content', 'task-artifacts']
[fetches 2024-01-17T11:35:11.657Z] attempt 1/5
[fetches 2024-01-17T11:35:11.658Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RVfZDOWQQxyWrdjqg2b_ng/artifacts/public/build/geckodriver.tar.gz to /opt/worker/tasks/task_170549130489610/fetches/geckodriver.tar.gz
[fetches 2024-01-17T11:35:11.658Z] attempt 1/5attempt 1/5
[fetches 2024-01-17T11:35:11.658Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bftZNfrPRGeRDzoTDnXbGA/artifacts/public/build/target.condprof.tests.tar.gz to /opt/worker/tasks/task_170549130489610/fetches/target.condprof.tests.tar.gz
[fetches 2024-01-17T11:35:11.658Z] 
[fetches 2024-01-17T11:35:11.658Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TXOkBzn-SNGGoCivlo2psQ/artifacts/public/build/python.tar.zst to /opt/worker/tasks/task_170549130489610/fetches/python.tar.zst
[fetches 2024-01-17T11:35:11.658Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RVfZDOWQQxyWrdjqg2b_ng/artifacts/public/build/geckodriver.tar.gz
[fetches 2024-01-17T11:35:11.658Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bftZNfrPRGeRDzoTDnXbGA/artifacts/public/build/target.condprof.tests.tar.gz
[fetches 2024-01-17T11:35:11.658Z] attempt 1/5
[fetches 2024-01-17T11:35:11.658Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bftZNfrPRGeRDzoTDnXbGA/artifacts/public/build/target.dmg to /opt/worker/tasks/task_170549130489610/fetches/target.dmg
[fetches 2024-01-17T11:35:11.658Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TXOkBzn-SNGGoCivlo2psQ/artifacts/public/build/python.tar.zst
[fetches 2024-01-17T11:35:11.659Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bftZNfrPRGeRDzoTDnXbGA/artifacts/public/build/target.dmg
[fetches 2024-01-17T11:35:11.659Z] fetch-content:188: DeprecationWarning: cafile, capath and cadefault are deprecated, use a custom context instead.
[fetches 2024-01-17T11:35:11.659Z]   ) if certifi else urllib.request.urlopen(url, timeout=60) as fh:
[fetches 2024-01-17T11:35:12.027Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bftZNfrPRGeRDzoTDnXbGA/artifacts/public/build/target.condprof.tests.tar.gz resolved to 212984 bytes with sha256 d13b7902c24b863fddd773b2301a0d8bb39c83c2608bc29379911c3997b2b9f1 in 0.369s
[fetches 2024-01-17T11:35:12.028Z] Extracting /opt/worker/tasks/task_170549130489610/fetches/target.condprof.tests.tar.gz to /opt/worker/tasks/task_170549130489610/fetches
[fetches 2024-01-17T11:35:12.134Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RVfZDOWQQxyWrdjqg2b_ng/artifacts/public/build/geckodriver.tar.gz resolved to 2242462 bytes with sha256 756c2154d323969773aab998906eabfcc52c59e032cd25949b4bbaa3b20bcab7 in 0.476s
[fetches 2024-01-17T11:35:12.143Z] Extracting /opt/worker/tasks/task_170549130489610/fetches/geckodriver.tar.gz to /opt/worker/tasks/task_170549130489610/fetches
[fetches 2024-01-17T11:35:12.185Z] /opt/worker/tasks/task_170549130489610/fetches/target.condprof.tests.tar.gz extracted in 0.157s
[fetches 2024-01-17T11:35:12.185Z] Removing /opt/worker/tasks/task_170549130489610/fetches/target.condprof.tests.tar.gz
[fetches 2024-01-17T11:35:12.268Z] /opt/worker/tasks/task_170549130489610/fetches/geckodriver.tar.gz extracted in 0.125s
[fetches 2024-01-17T11:35:12.269Z] Removing /opt/worker/tasks/task_170549130489610/fetches/geckodriver.tar.gz
[fetches 2024-01-17T11:35:13.017Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TXOkBzn-SNGGoCivlo2psQ/artifacts/public/build/python.tar.zst resolved to 48102376 bytes with sha256 93d9408c0aded41b86281aabc2b7476e132f5d637c7156927db3dc05a2e1f23e in 1.358s
[fetches 2024-01-17T11:35:13.021Z] Extracting /opt/worker/tasks/task_170549130489610/fetches/python.tar.zst to /opt/worker/tasks/task_170549130489610/fetches
[fetches 2024-01-17T11:35:14.141Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bftZNfrPRGeRDzoTDnXbGA/artifacts/public/build/target.dmg resolved to 155297511 bytes with sha256 077acb8bb32a9c78bf1536ccc718558bc0b93857c6429cfbba064323ec589349 in 2.482s
[fetches 2024-01-17T11:35:14.914Z] /opt/worker/tasks/task_170549130489610/fetches/python.tar.zst extracted in 1.893s
[fetches 2024-01-17T11:35:14.914Z] Removing /opt/worker/tasks/task_170549130489610/fetches/python.tar.zst
[fetches 2024-01-17T11:35:14.915Z] PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 3.2894094860000003, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-01-17T11:35:14.930Z] finished fetching artifacts
[setup 2024-01-17T11:35:14.930Z] Setting up local python environment
[setup 2024-01-17T11:35:14.930Z] updated PATH with python artifact: /opt/worker/tasks/task_170549130489610/fetches/python/bin:/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin
[task 2024-01-17T11:35:14.930Z] executing ['bash', '-cx', 'python3 -m venv . && bin/python3 condprof/main.py  --force-new  --geckodriver ${MOZ_FETCHES_DIR}/geckodriver --firefox ${MOZ_FETCHES_DIR}/target.dmg --scenario settled ../../archive']in fetches/condprofile
[task 2024-01-17T11:35:14.933Z] + python3 -m venv .
[task 2024-01-17T11:35:17.260Z] + bin/python3 condprof/main.py --force-new --geckodriver /opt/worker/tasks/task_170549130489610/fetches/geckodriver --firefox /opt/worker/tasks/task_170549130489610/fetches/target.dmg --scenario settled ../../archive
[task 2024-01-17T11:35:17.594Z] Looking in links: https://pypi.pub.build.mozilla.org/pub/, https://pypi.pub.build.mozilla.org/pub/
[task 2024-01-17T11:35:18.522Z] ERROR: Could not find a version that satisfies the requirement aiohttp==3.7.4 (from versions: 3.5.4, 3.7.4.post0)
[task 2024-01-17T11:35:18.522Z] ERROR: No matching distribution found for aiohttp==3.7.4
[task 2024-01-17T11:35:18.582Z] Traceback (most recent call last):
[task 2024-01-17T11:35:18.582Z]   File "/opt/worker/tasks/task_170549130489610/fetches/condprofile/condprof/../condprof/check_install.py", line 20, in install_reqs
[task 2024-01-17T11:35:18.582Z]     import yaml  # NOQA
[task 2024-01-17T11:35:18.582Z] ModuleNotFoundError: No module named 'yaml'
[task 2024-01-17T11:35:18.582Z] 
[task 2024-01-17T11:35:18.582Z] During handling of the above exception, another exception occurred:
[task 2024-01-17T11:35:18.582Z] 
[task 2024-01-17T11:35:18.582Z] Traceback (most recent call last):
[task 2024-01-17T11:35:18.582Z]   File "condprof/main.py", line 17, in <module>
[task 2024-01-17T11:35:18.582Z]     check()
[task 2024-01-17T11:35:18.582Z]   File "/opt/worker/tasks/task_170549130489610/fetches/condprofile/condprof/../condprof/check_install.py", line 69, in check
[task 2024-01-17T11:35:18.582Z]     if install_reqs():
[task 2024-01-17T11:35:18.582Z]   File "/opt/worker/tasks/task_170549130489610/fetches/condprofile/condprof/../condprof/check_install.py", line 46, in install_reqs
[task 2024-01-17T11:35:18.582Z]     subprocess.check_call(
[task 2024-01-17T11:35:18.582Z]   File "/opt/worker/tasks/task_170549130489610/fetches/python/lib/python3.8/subprocess.py", line 364, in check_call
[task 2024-01-17T11:35:18.582Z]     raise CalledProcessError(retcode, cmd)
[task 2024-01-17T11:35:18.582Z] subprocess.CalledProcessError: Command '['/opt/worker/tasks/task_170549130489610/fetches/condprofile/bin/python3', '-m', 'pip', 'install', '--no-cache-dir', '--isolated', '--find-links', 'https://pypi.pub.build.mozilla.org/pub/', 'aiohttp==3.7.4']' returned non-zero exit status 1.
[taskcluster 2024-01-17T11:35:18.592Z]    Exit Code: 1
[taskcluster 2024-01-17T11:35:18.592Z]    User Time: 4.345869s
[taskcluster 2024-01-17T11:35:18.592Z]  Kernel Time: 2.215977s
[taskcluster 2024-01-17T11:35:18.592Z]    Wall Time: 9.763686s
[taskcluster 2024-01-17T11:35:18.592Z]       Result: FAILED
[taskcluster 2024-01-17T11:35:18.592Z] === Task Finished ===
[taskcluster 2024-01-17T11:35:18.592Z] Task Duration: 9.76949s
[taskcluster:error] Uploading error artifact public/condprof from file archive with message "Could not read directory '/opt/worker/tasks/task_170549130489610/archive'", reason "file-missing-on-worker" and expiry 2024-04-16T09:31:32.896Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/opt/worker/tasks/task_170549130489610/archive'
[taskcluster 2024-01-17T11:35:18.726Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-04-16T09:31:32.896Z
[taskcluster:error] exit status 1

Comment 1

[Intermittent Failures Robot]

3 failures in 3599 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• macosx1015-64-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875050&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 2

[Iulian Moraru]

This seems to be permafailing now. The last green run was here.

Hi Andrew! Can you please take a look at this?
Thank you!

Comment 3

[Intermittent Failures Robot]

23 failures in 4034 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 23

Platform and build breakdown:

• macosx1015-64-qr: 23
  • opt: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875050&startday=2024-01-22&endday=2024-01-28&tree=all

Comment 4

[Andrew Halberstadt [:ahal]]

Re-directing to perftest team.

Comment 5

[Greg Mierzwinski [:sparky]]

This is related to an update to python 3.8 on the mac machines. We'll need to add the 3.8 version of aiohttp to our internal pypi mirror.

Comment 6

[Greg Mierzwinski [:sparky]]

Wrong bug, sorry!

Comment 7

[Greg Mierzwinski [:sparky]]

Nvm, we have up to python 3.9 for the aiohttp module. I'm looking into it.

Comment 8

[Greg Mierzwinski [:sparky]]

I've got a potential fix for this in this try run: https://treeherder.mozilla.org/jobs?repo=try&revision=238698a0c9b9068fefdc4dc07de983468029fe92

Comment 9

[Intermittent Failures Robot]

28 failures in 3952 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 28

Platform and build breakdown:

• macosx1015-64-qr: 28
  • opt: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875050&startday=2024-01-29&endday=2024-02-04&tree=all

Comment 10

[Intermittent Failures Robot]

28 failures in 4009 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 28

Platform and build breakdown:

• macosx1015-64-qr: 28
  • opt: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875050&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 11

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1875050 - Fix, and reorder python reqs for condprof. r?jmaher

This patch reorders the python requirements to ensure that mozlog gets built first in CI. This will prevent pip from trying to find versions that don't exist in our internal pypi mirror when we are trying to install those versions from source. This patch also updates the name of the aiohttp req to better reflect what's found in our pypi mirror.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/753f8762281c
Fix, and reorder python reqs for condprof. r=jmaher

Comment 13

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/753f8762281c

Comment 14

[Intermittent Failures Robot]

1 failures in 3125 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875050&startday=2024-02-12&endday=2024-02-18&tree=all