1848881 - Perma [tier 2] macOS 13 browsertime-chromium - KeyError: 'test'

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: chorotan [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=426122558&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cBCNxvTGTeWTU1CK6D3T9A/runs/0/artifacts/public/logs/live_backing.log

---

[fetches 2023-08-15T21:50:21.380Z] Archive:  /opt/worker/tasks/task_1692136214/fetches/ffmpeg-macos.zip
[fetches 2023-08-15T21:50:21.380Z]    creating: ffmpeg-macos/
[fetches 2023-08-15T21:50:21.548Z]   inflating: ffmpeg-macos/ffmpeg     /opt/worker/tasks/task_1692136214/fetches/node.tar.zst extracted in 0.405s
[fetches 2023-08-15T21:50:21.548Z] Removing /opt/worker/tasks/task_1692136214/fetches/node.tar.zst
[fetches 2023-08-15T21:50:21.817Z] 
[fetches 2023-08-15T21:50:21.818Z]   inflating: __MACOSX/ffmpeg-macos/._ffmpeg  
[fetches 2023-08-15T21:50:22.242Z]   inflating: ffmpeg-macos/ffprobe    
[fetches 2023-08-15T21:50:22.242Z]   inflating: __MACOSX/ffmpeg-macos/._ffprobe  
[fetches 2023-08-15T21:50:22.242Z] /opt/worker/tasks/task_1692136214/fetches/ffmpeg-macos.zip extracted in 0.872s
[fetches 2023-08-15T21:50:22.242Z] Removing /opt/worker/tasks/task_1692136214/fetches/ffmpeg-macos.zip
[fetches 2023-08-15T21:50:22.524Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cLYDXgdnSqSTo8fx0QvYtQ/artifacts/public/build/car_macosx.tar.zst resolved to 118369472 bytes with sha256 e83c4a19ae3127db2b755ba6bbf5f417595ba0013133c69ffdc6ae4c3ff9821c in 3.806s
[fetches 2023-08-15T21:50:22.525Z] Extracting /opt/worker/tasks/task_1692136214/fetches/car_macosx.tar.zst to /opt/worker/tasks/task_1692136214/fetches
[fetches 2023-08-15T21:50:23.006Z] /opt/worker/tasks/task_1692136214/fetches/car_macosx.tar.zst extracted in 0.481s
[fetches 2023-08-15T21:50:23.006Z] Removing /opt/worker/tasks/task_1692136214/fetches/car_macosx.tar.zst
[fetches 2023-08-15T21:50:23.235Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aLWQT1RUSL6DGDSKTfgaIA/artifacts/public/chromium-mac.tar.bz2 resolved to 123442229 bytes with sha256 41df9ded14aa06a3ca8681fd3ed41d93168446ea1d995524b61baceee4ef3f0a in 4.516s
[fetches 2023-08-15T21:50:23.236Z] Extracting /opt/worker/tasks/task_1692136214/fetches/chromium-mac.tar.bz2 to /opt/worker/tasks/task_1692136214/fetches
[fetches 2023-08-15T21:50:29.985Z] /opt/worker/tasks/task_1692136214/fetches/chromium-mac.tar.bz2 extracted in 6.749s
[fetches 2023-08-15T21:50:29.985Z] Removing /opt/worker/tasks/task_1692136214/fetches/chromium-mac.tar.bz2
[fetches 2023-08-15T21:50:29.986Z] PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 11.268835791, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2023-08-15T21:50:29.994Z] finished fetching artifacts
[task 2023-08-15T21:50:29.995Z] executing ['/usr/local/bin/python3', '-u', 'mozharness/scripts/raptor_script.py', '--cfg', 'mozharness/configs/raptor/mac_config.py', '--conditioned-profile=settled', '--extra-profiler-run', '--app=custom-car', '--project=mozilla-central', '--browsertime-browsertimejs', '$MOZ_FETCHES_DIR/browsertime/node_modules/browsertime/bin/browsertime.js', '--browsertime-node', '$MOZ_FETCHES_DIR/node/bin/node', '--browsertime-geckodriver', '$MOZ_FETCHES_DIR/geckodriver', '--browsertime-chromedriver', '$MOZ_FETCHES_DIR/chrome-mac/chromedriver', '--browsertime-ffmpeg', '$MOZ_FETCHES_DIR/ffmpeg-macos/ffmpeg', '--download-symbols', 'ondemand']
[task 2023-08-15T21:50:30.089Z] 21:50:30     INFO - ConsoleLogger online at 20230815 21:50:30Z in /opt/worker/tasks/task_1692136214
[task 2023-08-15T21:50:30.089Z] 21:50:30     INFO - Run as mozharness/scripts/raptor_script.py --cfg mozharness/configs/raptor/mac_config.py --conditioned-profile=settled --extra-profiler-run --app=custom-car --project=mozilla-central --browsertime-browsertimejs $MOZ_FETCHES_DIR/browsertime/node_modules/browsertime/bin/browsertime.js --browsertime-node $MOZ_FETCHES_DIR/node/bin/node --browsertime-geckodriver $MOZ_FETCHES_DIR/geckodriver --browsertime-chromedriver $MOZ_FETCHES_DIR/chrome-mac/chromedriver --browsertime-ffmpeg $MOZ_FETCHES_DIR/ffmpeg-macos/ffmpeg --download-symbols ondemand
[task 2023-08-15T21:50:30.094Z] Traceback (most recent call last):
[task 2023-08-15T21:50:30.094Z]   File "/opt/worker/tasks/task_1692136214/mozharness/scripts/raptor_script.py", line 19, in <module>
[task 2023-08-15T21:50:30.094Z]     raptor = Raptor()
[task 2023-08-15T21:50:30.094Z]   File "/opt/worker/tasks/task_1692136214/mozharness/mozharness/mozilla/testing/raptor.py", line 683, in __init__
[task 2023-08-15T21:50:30.094Z]     self.test = self.config["test"]
[task 2023-08-15T21:50:30.094Z] KeyError: 'test'
[taskcluster 2023-08-15T21:50:30.103Z]    Exit Code: 1
[taskcluster 2023-08-15T21:50:30.103Z]    User Time: 8.867755s
[taskcluster 2023-08-15T21:50:30.103Z]  Kernel Time: 2.248106s
[taskcluster 2023-08-15T21:50:30.103Z]    Wall Time: 11.470336s
[taskcluster 2023-08-15T21:50:30.103Z]       Result: FAILED
[taskcluster 2023-08-15T21:50:30.103Z] === Task Finished ===
[taskcluster 2023-08-15T21:50:30.103Z] Task Duration: 11.472308s
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory '/opt/worker/tasks/task_1692136214/build/blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2023-11-13T21:38:47.438Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/opt/worker/tasks/task_1692136214/build/blobber_upload_dir'
[taskcluster 2023-08-15T21:50:31.403Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cBCNxvTGTeWTU1CK6D3T9A/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2023-11-13T21:38:47.438Z
[taskcluster:error] exit status 1
[taskcluster:error] file-missing-on-worker: Could not read directory '/opt/worker/tasks/task_1692136214/build/blobber_upload_dir'

Comment 1

[Cosmin Sabou [:CosminS]]

The actual failure is here:

[fetches 2023-08-15T23:05:37.710Z] finished fetching artifacts
[task 2023-08-15T23:05:37.710Z] executing ['/usr/local/bin/python3', '-u', 'mozharness/scripts/raptor_script.py', '--cfg', 'mozharness/configs/raptor/mac_config.py', '--conditioned-profile=settled', '--extra-profiler-run', '--app=custom-car', '--project=mozilla-central', '--browsertime-browsertimejs', '$MOZ_FETCHES_DIR/browsertime/node_modules/browsertime/bin/browsertime.js', '--browsertime-node', '$MOZ_FETCHES_DIR/node/bin/node', '--browsertime-geckodriver', '$MOZ_FETCHES_DIR/geckodriver', '--browsertime-chromedriver', '$MOZ_FETCHES_DIR/chrome-mac/chromedriver', '--browsertime-ffmpeg', '$MOZ_FETCHES_DIR/ffmpeg-macos/ffmpeg', '--download-symbols', 'ondemand']
[task 2023-08-15T23:05:37.831Z] 23:05:37     INFO - ConsoleLogger online at 20230815 23:05:37Z in /opt/worker/tasks/task_1692140712
[task 2023-08-15T23:05:37.831Z] 23:05:37     INFO - Run as mozharness/scripts/raptor_script.py --cfg mozharness/configs/raptor/mac_config.py --conditioned-profile=settled --extra-profiler-run --app=custom-car --project=mozilla-central --browsertime-browsertimejs $MOZ_FETCHES_DIR/browsertime/node_modules/browsertime/bin/browsertime.js --browsertime-node $MOZ_FETCHES_DIR/node/bin/node --browsertime-geckodriver $MOZ_FETCHES_DIR/geckodriver --browsertime-chromedriver $MOZ_FETCHES_DIR/chrome-mac/chromedriver --browsertime-ffmpeg $MOZ_FETCHES_DIR/ffmpeg-macos/ffmpeg --download-symbols ondemand
[task 2023-08-15T23:05:37.836Z] Traceback (most recent call last):
[task 2023-08-15T23:05:37.836Z]   File "/opt/worker/tasks/task_1692140712/mozharness/scripts/raptor_script.py", line 19, in <module>
[task 2023-08-15T23:05:37.836Z]     raptor = Raptor()
[task 2023-08-15T23:05:37.836Z]   File "/opt/worker/tasks/task_1692140712/mozharness/mozharness/mozilla/testing/raptor.py", line 683, in __init__
[task 2023-08-15T23:05:37.836Z]     self.test = self.config["test"]
[task 2023-08-15T23:05:37.836Z] KeyError: 'test'
[taskcluster 2023-08-15T23:05:37.845Z]    Exit Code: 1

It first ran on this merge coming from the changes in Bug 1844638 where this platform was added.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1835065

Comment 3

[Andrej (:aglavic)]

Narrowed down the cause of this.
This is due to the way that the yml is defined and parsed.
When we define in the test set that we want to run the benchmark tests on the M2s it is assumed we want to run at least 1 job on an app at a minimum so when we leave chromium, safari, and CAR as "[]" for the M2 macs the code parsing is not ready for that and ends up passing an empty string and that is why we end up with a key-error for test.
There are 2 ways of resolving this, the first is to just create a new job that very closely mirrors the benchmark section with just chrome and firefox (the test we want run on the M2s) we can alternatively resolve the section of the parsing.
Will be looking into this later today but wanted to give an update

Comment 4

[Andrej (:aglavic)]

Leaving on need info to keep track of bug

Comment 5

[Intermittent Failures Robot]

11 failures in 3076 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10
• try: 1

Platform and build breakdown:

• macosx1300-64-shippable-qr: 10
  • opt: 10
• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1848881&startday=2023-08-14&endday=2023-08-20&tree=all

Comment 6

[Intermittent Failures Robot]

16 failures in 2218 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16

Platform and build breakdown:

• macosx1300-64-shippable-qr: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1848881&startday=2023-08-21&endday=2023-08-27&tree=all

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1844638

Comment 8

[Intermittent Failures Robot]

15 failures in 3019 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 15

Platform and build breakdown:

• macosx1300-64-shippable-qr: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1848881&startday=2023-08-28&endday=2023-09-03&tree=all

Comment 9

[Intermittent Failures Robot]

10 failures in 3267 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10

Platform and build breakdown:

• macosx1300-64-shippable-qr: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1848881&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 10

[Andrej (:aglavic)]

Attachment: Bug 1848881 - Make test a required value when compiling a list of tests from the try and CI yml files. r?#perftest

Comment 11

[Andrej (:aglavic)]

Attachment: Bug 1848881 - Make test a required value when compiling a list of tests from the try and CI yml files. r?#perftest

Comment 12

[Intermittent Failures Robot]

12 failures in 3337 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12

Platform and build breakdown:

• macosx1300-64-shippable-qr: 12
  • opt: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1848881&startday=2023-09-11&endday=2023-09-17&tree=all

Comment 13

[Intermittent Failures Robot]

9 failures in 3850 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 9

Platform and build breakdown:

• macosx1300-64-shippable-qr: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1848881&startday=2023-09-18&endday=2023-09-24&tree=all

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7095adada046
Make test a required value when compiling a list of tests from the try and CI yml files. r=perftest-reviewers,taskgraph-reviewers,gbrown,sparky

Comment 15

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/7095adada046

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:aglavic, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox119 to wontfix.

For more information, please visit BugBot documentation.

Comment 17

[Andrej (:aglavic)]

This isn't important enough to need an uplift

Comment 18

[Intermittent Failures Robot]

3 failures in 3726 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• macosx1300-64-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1848881&startday=2023-09-25&endday=2023-10-01&tree=all