1884354 - Frequent subprocess.CalledProcessError: Command '['git', 'clone', '-c', 'http.postBuffer=2147483648', '-c', 'core.autocrlf=false', 'https://github.com/mozilla/perf-automation] returned non-zero exit status 128.

Status: RESOLVED INCOMPLETE

(Testing :: Raptor, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ctuns [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=450047826&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WVCljwO2QcWogK2QUItElQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-03-08T11:15:22.739Z] 11:15:22     INFO -  raptor-main Info: matrix-react-bench
[task 2024-03-08T11:15:22.739Z] 11:15:22     INFO -  raptor-perftest Info: Using an empty profile.
[task 2024-03-08T11:15:22.740Z] 11:15:22     INFO -  raptor-perftest Warning: [chrome] Failed to get browser meta data through mozversion: LocalAppNotFoundError-Application not found at: /opt/google/chrome
[task 2024-03-08T11:15:22.740Z] 11:15:22     INFO -  raptor-perftest Info: Attempting to get version through fallback method...
[task 2024-03-08T11:15:22.802Z] 11:15:22     INFO -  raptor-perftest Info: Browser name: chrome
[task 2024-03-08T11:15:22.803Z] 11:15:22     INFO -  raptor-perftest Info: Browser version: 122.0.6261.111
[task 2024-03-08T11:15:22.803Z] 11:15:22     INFO -  raptor-perftest Warning: [chrome] Failed to get browser meta data through mozversion: LocalAppNotFoundError-Application not found at: /opt/google/chrome
[task 2024-03-08T11:15:22.803Z] 11:15:22     INFO -  raptor-perftest Info: Attempting to get version through fallback method...
[task 2024-03-08T11:15:22.828Z] 11:15:22     INFO -  raptor-perftest Info: Browser name: chrome
[task 2024-03-08T11:15:22.828Z] 11:15:22     INFO -  raptor-perftest Info: Browser version: 122.0.6261.111
[task 2024-03-08T11:15:22.829Z] 11:15:22     INFO -  raptor-perftest Info: Post startup delay set to 30000 ms
[task 2024-03-08T11:15:22.831Z] 11:15:22     INFO -  raptor-perftest Info: main raptor init, config is: {'app': 'chrome', 'binary': '/usr/bin/google-chrome', 'platform': 'linux', 'processor': 'x86_64', 'run_local': False, 'obj_path': None, 'gecko_profile': False, 'gecko_profile_interval': None, 'gecko_profile_entries': None, 'gecko_profile_extra_threads': [], 'gecko_profile_threads': None, 'gecko_profile_features': None, 'extra_profiler_run': True, 'symbols_path': 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FsYusge8QLKspdrl1moc-g/artifacts/public/build/target.crashreporter-symbols.zip', 'host': '127.0.0.1', 'cold': False, 'live_sites': False, 'is_release_build': False, 'e10s': True, 'device_name': None, 'fission': True, 'disable_perf_tuning': False, 'conditioned_profile': None, 'test_bytecode_cache': False, 'chimera': False, 'extra_prefs': {'fission.autostart': True}, 'environment': {}, 'project': 'mozilla-central', 'verbose': False, 'extra_summary_methods': [], 'benchmark_repository': None, 'benchmark_revision': None, 'benchmark_branch': None, 'clean': False, 'screenshot_on_failure': True}
[task 2024-03-08T11:15:22.831Z] 11:15:22     INFO -  raptor-browsertime Info: cwd: '/home/cltbld/tasks/task_170989642618752/build'
[task 2024-03-08T11:15:22.831Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_node: /home/cltbld/tasks/task_170989642618752/fetches/node/bin/node
[task 2024-03-08T11:15:22.832Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_node: os.stat_result(st_mode=33261, st_ino=9360671, st_dev=64768, st_nlink=1, st_uid=1006, st_gid=1007, st_size=92930512, st_atime=1709896430, st_mtime=1708940159, st_ctime=1709896430)
[task 2024-03-08T11:15:22.832Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_browsertimejs: /home/cltbld/tasks/task_170989642618752/fetches/browsertime/node_modules/browsertime/bin/browsertime.js
[task 2024-03-08T11:15:22.833Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_browsertimejs: os.stat_result(st_mode=33261, st_ino=9329330, st_dev=64768, st_nlink=1, st_uid=1006, st_gid=1007, st_size=7076, st_atime=1709896430, st_mtime=1708958795, st_ctime=1709896430)
[task 2024-03-08T11:15:22.833Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_geckodriver: /home/cltbld/tasks/task_170989642618752/fetches/geckodriver
[task 2024-03-08T11:15:22.834Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_geckodriver: os.stat_result(st_mode=33261, st_ino=9322545, st_dev=64768, st_nlink=1, st_uid=1006, st_gid=1007, st_size=9794056, st_atime=1709896430, st_mtime=1707775093, st_ctime=1709896430)
[task 2024-03-08T11:15:22.834Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_chromedriver: /home/cltbld/tasks/task_170989642618752/fetches/{}chromedriver
[task 2024-03-08T11:15:22.835Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_chromedriver: [Errno 2] No such file or directory: '/home/cltbld/tasks/task_170989642618752/fetches/{}chromedriver'
[task 2024-03-08T11:15:22.835Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_user_args: []
[task 2024-03-08T11:15:22.835Z] 11:15:22     INFO -  raptor-browsertime Info: browsertime_user_args: stat: path should be string, bytes, os.PathLike or integer, not list
[task 2024-03-08T11:15:22.836Z] 11:15:22     INFO -  raptor-perftest Info: starting test: matrix-react-bench
[task 2024-03-08T11:15:22.836Z] 11:15:22     INFO -  raptor-benchmark Info: Cloning the benchmarks to /tmp/tmpbmssby4g/performance-tests/perf-automation
[task 2024-03-08T11:15:22.836Z] 11:15:22     INFO -  Cloning into '/tmp/tmpbmssby4g/performance-tests/perf-automation'...
[task 2024-03-08T11:17:03.448Z] 11:17:03     INFO -  error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
[task 2024-03-08T11:17:03.449Z] 11:17:03     INFO -  error: 1473 bytes of body are still expected
[task 2024-03-08T11:17:03.450Z] 11:17:03     INFO -  fetch-pack: unexpected disconnect while reading sideband packet
[task 2024-03-08T11:17:03.450Z] 11:17:03     INFO -  fatal: early EOF
[task 2024-03-08T11:17:03.450Z] 11:17:03     INFO -  fatal: fetch-pack: invalid index-pack output
[task 2024-03-08T11:17:03.454Z] 11:17:03    ERROR -  Traceback (most recent call last):
[task 2024-03-08T11:17:03.455Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/raptor.py", line 188, in <module>
[task 2024-03-08T11:17:03.455Z] 11:17:03     INFO -      main()
[task 2024-03-08T11:17:03.455Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/raptor.py", line 137, in main
[task 2024-03-08T11:17:03.455Z] 11:17:03     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2024-03-08T11:17:03.456Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/perftest.py", line 500, in run_tests
[task 2024-03-08T11:17:03.456Z] 11:17:03     INFO -      self.run_test(test, timeout=int(test.get("page_timeout")))
[task 2024-03-08T11:17:03.456Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/browsertime/base.py", line 910, in run_test
[task 2024-03-08T11:17:03.456Z] 11:17:03     INFO -      self.run_test_setup(test)
[task 2024-03-08T11:17:03.457Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/browsertime/base.py", line 156, in run_test_setup
[task 2024-03-08T11:17:03.457Z] 11:17:03     INFO -      self.benchmark = Benchmark(self.config, test, debug_mode=self.debug_mode)
[task 2024-03-08T11:17:03.457Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/benchmark.py", line 42, in __init__
[task 2024-03-08T11:17:03.457Z] 11:17:03     INFO -      self.setup_benchmarks(
[task 2024-03-08T11:17:03.458Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/benchmark.py", line 350, in setup_benchmarks
[task 2024-03-08T11:17:03.458Z] 11:17:03     INFO -      bench_dir = self._setup_git_benchmarks(
[task 2024-03-08T11:17:03.458Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/benchmark.py", line 271, in _setup_git_benchmarks
[task 2024-03-08T11:17:03.459Z] 11:17:03     INFO -      self._full_clone(benchmark_repository, external_repo_path)
[task 2024-03-08T11:17:03.459Z] 11:17:03     INFO -    File "/home/cltbld/tasks/task_170989642618752/build/tests/raptor/raptor/benchmark.py", line 104, in _full_clone
[task 2024-03-08T11:17:03.459Z] 11:17:03     INFO -      subprocess.check_call(
[task 2024-03-08T11:17:03.459Z] 11:17:03     INFO -    File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
[task 2024-03-08T11:17:03.460Z] 11:17:03     INFO -      raise CalledProcessError(retcode, cmd)
[task 2024-03-08T11:17:03.460Z] 11:17:03     INFO -  subprocess.CalledProcessError: Command '['git', 'clone', '-c', 'http.postBuffer=2147483648', '-c', 'core.autocrlf=false', 'https://github.com/mozilla/perf-automation', '/tmp/tmpbmssby4g/performance-tests/perf-automation']' returned non-zero exit status 128.
[task 2024-03-08T11:17:03.507Z] 11:17:03     INFO - Return code: 1
[task 2024-03-08T11:17:03.508Z] 11:17:03  WARNING - setting return code to 1
[task 2024-03-08T11:17:03.508Z] 11:17:03     INFO - Copying Raptor results to upload dir:
[task 2024-03-08T11:17:03.508Z] 11:17:03     INFO - /home/cltbld/tasks/task_170989642618752/build/blobber_upload_dir/perfherder-data.json
[task 2024-03-08T11:17:03.508Z] 11:17:03     INFO - Copying raptor results from /home/cltbld/tasks/task_170989642618752/build/raptor.json to /home/cltbld/tasks/task_170989642618752/build/blobber_upload_dir/perfherder-data.json
[task 2024-03-08T11:17:03.508Z] 11:17:03 CRITICAL - Error copying results /home/cltbld/tasks/task_170989642618752/build/raptor.json to upload dir /home/cltbld/tasks/task_170989642618752/build/blobber_upload_dir/perfherder-data.json
[task 2024-03-08T11:17:03.508Z] 11:17:03     INFO - [Errno 2] No such file or directory: '/home/cltbld/tasks/task_170989642618752/build/raptor.json'
[task 2024-03-08T11:17:03.509Z] 11:17:03     INFO - Running post-action listener: _package_coverage_data
[task 2024-03-08T11:17:03.509Z] 11:17:03     INFO - Running post-action listener: _resource_record_post_action
[task 2024-03-08T11:17:03.509Z] 11:17:03     INFO - Running post-action listener: process_java_coverage_data
[task 2024-03-08T11:17:03.509Z] 11:17:03     INFO - Running post-action listener: stop_device
[task 2024-03-08T11:17:03.509Z] 11:17:03     INFO - [mozharness: 2024-03-08 11:17:03.509279Z] Finished run-tests step (success)
[task 2024-03-08T11:17:03.509Z] 11:17:03     INFO - Running post-run listener: _resource_record_post_run
[task 2024-03-08T11:17:03.669Z] 11:17:03     INFO - Total resource usage - Wall time: 101s; CPU: 1%; Read bytes: 86261760; Write bytes: 80281600; Read time: 144; Write time: 928
[task 2024-03-08T11:17:03.669Z] 11:17:03     INFO - TinderboxPrint: CPU usage<br/>0.9%
[task 2024-03-08T11:17:03.669Z] 11:17:03     INFO - TinderboxPrint: I/O read bytes / time<br/>86,261,760 / 144
[task 2024-03-08T11:17:03.669Z] 11:17:03     INFO - TinderboxPrint: I/O write bytes / time<br/>80,281,600 / 928
[task 2024-03-08T11:17:03.669Z] 11:17:03     INFO - TinderboxPrint: CPU idle<br/>801.9 (99.0%)
[task 2024-03-08T11:17:03.669Z] 11:17:03     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2024-03-08T11:17:03.670Z] 11:17:03     INFO - install-chromium-distribution - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-03-08T11:17:03.704Z] 11:17:03     INFO - run-tests - Wall time: 101s; CPU: 1%; Read bytes: 86261760; Write bytes: 80281600; Read time: 144; Write time: 928
[task 2024-03-08T11:17:04.366Z] 11:17:04  WARNING - returning nonzero exit status 1
[taskcluster 2024-03-08T11:17:04.414Z]    Exit Code: 1
[taskcluster 2024-03-08T11:17:04.414Z]    User Time: 1m25.18361s
[taskcluster 2024-03-08T11:17:04.414Z]  Kernel Time: 7.870919s
[taskcluster 2024-03-08T11:17:04.414Z]    Wall Time: 3m14.879118454s
[taskcluster 2024-03-08T11:17:04.414Z]       Result: FAILED
[taskcluster 2024-03-08T11:17:04.414Z] === Task Finished ===
[taskcluster 2024-03-08T11:17:04.414Z] Task Duration: 3m14.881087506s
[taskcluster 2024-03-08T11:17:04.533Z] Uploading artifact public/logs/localconfig.json from file logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2024-06-06T11:11:44.690Z
[taskcluster 2024-03-08T11:17:04.821Z] Uploading artifact public/test_info/profile_resource-usage.json from file build/blobber_upload_dir/profile_resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-06-06T11:11:44.690Z
[taskcluster 2024-03-08T11:17:05.100Z] Uploading artifact public/test_info/resource-usage.json from file build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-06-06T11:11:44.690Z
[taskcluster 2024-03-08T11:17:05.380Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-06-06T11:11:44.690Z
[taskcluster:error] exit status 1

Comment 1

[Cristian Tuns]

Hi, can you please take a look at this?
https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=browser%2Ctime&revision=ac5529a784bdd0522e58c4bcdb3293674601ac52&selectedTaskRun=SixBjjkQTQWWzSyN3qNP0A.0

Comment 4

[Intermittent Failures Robot]

34 failures in 659 pushes (0.052 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 7
• mozilla-beta: 1
• autoland: 26

Platform and build breakdown:

• linux1804-64-shippable-qr: 16
  • opt: 16
• windows10-64-shippable-qr: 6
  • opt: 6
• macosx1015-64-shippable-qr: 6
  • opt: 6
• linux1804-64-nightlyasrelease-qr: 2
  • opt: 2
• windows10-64-nightlyasrelease-qr: 2
  • opt: 2
• macosx1015-64-nightlyasrelease-qr: 1
  • opt: 1
• macosx1400-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-03-08&endday=2024-03-08&tree=all

Comment 5

[Intermittent Failures Robot]

35 failures in 4198 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 8
• autoland: 26
• mozilla-beta: 1

Platform and build breakdown:

• macosx1015-64-shippable-qr: 7
  • opt: 7
• linux1804-64-shippable-qr: 16
  • opt: 16
• linux1804-64-nightlyasrelease-qr: 2
  • opt: 2
• windows10-64-shippable-qr: 6
  • opt: 6
• windows10-64-nightlyasrelease-qr: 2
  • opt: 2
• macosx1015-64-nightlyasrelease-qr: 1
  • opt: 1
• macosx1400-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-03-04&endday=2024-03-10&tree=all

Comment 6

[Kash Shampur [:kshampur] ⌚EST]

I don't believe there is much we can do, this seems to be a network issue that pops up every now and then e.g. Bug 1832583
For now it looks like it has subsided

Though maybe we can consider looking into increasing the buffer. I will file a bug for that

Comment 7

[Intermittent Failures Robot]

1 failures in 3849 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-03-11&endday=2024-03-17&tree=all

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 9

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=454957053&repo=autoland

Comment 10

[Intermittent Failures Robot]

18 failures in 143 pushes (0.126 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18

Platform and build breakdown:

• macosx1015-64-nightlyasrelease-qr: 1
  • opt: 1
• linux1804-64-nightlyasrelease-qr: 1
  • opt: 1
• linux1804-64-shippable-qr: 5
  • opt: 5
• macosx1015-64-shippable-qr: 5
  • opt: 5
• windows10-64-shippable-qr: 5
  • opt: 5
• windows10-64-nightlyasrelease-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-04-20&endday=2024-04-20&tree=all

Comment 11

[Intermittent Failures Robot]

19 failures in 4252 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 19

Platform and build breakdown:

• linux1804-64-shippable-qr: 6
  • opt: 6
• macosx1015-64-nightlyasrelease-qr: 1
  • opt: 1
• linux1804-64-nightlyasrelease-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 5
  • opt: 5
• windows10-64-shippable-qr: 5
  • opt: 5
• windows10-64-nightlyasrelease-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-04-15&endday=2024-04-21&tree=all

Comment 12

[Intermittent Failures Robot]

4 failures in 3649 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform and build breakdown:

• macosx1015-64-nightlyasrelease-qr: 1
  • opt: 1
• linux1804-64-shippable-qr: 1
  • opt: 1
• windows10-64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-05-06&endday=2024-05-12&tree=all

Comment 13

[Cristian Tuns]

This started to be an issue again, starting with this merge: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=browsertime&revision=80ae2c7908e4cc558336a6f6a78097d9c2c9317d&selectedTaskRun=SGOc_-oOSVW337Zm5VMXIA.0

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Kash, does this need a chromedriver update? The ticket for 125 says that landed and 126 would be due in early June.

Comment 15

[Kash Shampur [:kshampur] ⌚EST]

That chromedriver warning is a bit misleading unfortunately (and it appears even on passing chrome tasks). To answer your question an update isn't needed here. versions 124 and 125 chrome drivers were correctly fetched. The issue is still just the git checkout

anyhow, :aglavic is working on Bug 1804694 which should hopefully help with this

Comment 16

[Greg Mierzwinski [:sparky]]

For context, that warning happens because we try getting a version using mozversion first, and then a custom method after that. We're hoping someone adds something for gathering the version number of those browsers to mozversion someday (might need to be us who does that :).

Comment 17

[Intermittent Failures Robot]

15 failures in 3995 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 7

Platform and build breakdown:

• macosx1015-64-nightlyasrelease-qr: 1
  • opt: 1
• linux1804-64-shippable-qr: 5
  • opt: 5
• windows10-64-shippable-qr: 4
  • opt: 4
• macosx1015-64-shippable-qr: 3
  • opt: 3
• windows10-64-nightlyasrelease-qr: 1
  • opt: 1
• android-hw-p6-13-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-05-20&endday=2024-05-26&tree=all

Comment 18

[Intermittent Failures Robot]

19 failures in 851 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19

Platform and build breakdown:

• macosx1015-64-nightlyasrelease-qr: 2
  • opt: 2
• windows10-64-nightlyasrelease-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 5
  • opt: 5
• linux1804-64-shippable-qr: 5
  • opt: 5
• linux1804-64-nightlyasrelease-qr: 1
  • opt: 1
• windows10-64-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-05-28&endday=2024-05-28&tree=all

Comment 19

[Intermittent Failures Robot]

19 failures in 4202 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 19

Platform and build breakdown:

• macosx1015-64-nightlyasrelease-qr: 2
  • opt: 2
• windows10-64-nightlyasrelease-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 5
  • opt: 5
• linux1804-64-shippable-qr: 5
  • opt: 5
• linux1804-64-nightlyasrelease-qr: 1
  • opt: 1
• windows10-64-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-05-27&endday=2024-06-02&tree=all

Comment 20

[Intermittent Failures Robot]

88 failures in 591 pushes (0.149 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 46
• autoland: 36
• mozilla-beta: 6

Platform and build breakdown:

• macosx1015-64-nightlyasrelease-qr: 3
  • opt: 3
• linux1804-64-shippable-qr: 26
  • opt: 26
• macosx1015-64-shippable-qr: 26
  • opt: 26
• windows10-64-shippable-qr: 26
  • opt: 26
• linux1804-64-nightlyasrelease-qr: 3
  • opt: 3
• windows10-64-nightlyasrelease-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-06-03&endday=2024-06-03&tree=all

Comment 21

[Intermittent Failures Robot]

110 failures in 3649 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #5 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 48
• mozilla-central: 56
• mozilla-beta: 6

Platform and build breakdown:

• windows10-64-shippable-qr: 28
  • opt: 28
• macosx1015-64-nightlyasrelease-qr: 3
  • opt: 3
• linux1804-64-shippable-qr: 35
  • opt: 35
• macosx1015-64-shippable-qr: 34
  • opt: 34
• linux1804-64-nightlyasrelease-qr: 3
  • opt: 3
• linux1804-64-clang-trunk-qr: 3
  • opt: 3
• windows10-64-nightlyasrelease-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-06-03&endday=2024-06-09&tree=all

Comment 22

[Intermittent Failures Robot]

12 failures in 4577 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12

Platform and build breakdown:

• windows10-64-nightlyasrelease-qr: 1
  • opt: 1
• windows10-64-shippable-qr: 5
  • opt: 5
• linux1804-64-shippable-qr: 5
  • opt: 5
• linux1804-64-nightlyasrelease-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-06-17&endday=2024-06-23&tree=all

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 24

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=485589417&repo=autoland

Comment 25

[Intermittent Failures Robot]

1 failures in 4722 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-hw-a55-14-0-aarch64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-12-02&endday=2024-12-08&tree=all

Comment 26

[Intermittent Failures Robot]

Summary

2 failures in 3774 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 1
  • opt: 1
• windows11-64-shippable-qr: 1
  • opt: 1

Table

	1proc	1proc-s	1proc-swr	a11y-checks	a11y-checks-s	a11y-checks-swr	a11y-checks-swr-s	condprof-s	fis	fis-hv	fis-hv-s	fis-s	headless	headless-s	http2	http3	mda-gpu	mda-gpu-s	msix	msix-s	no_variant	nofis	nofis-aab	nofis-s	nofis-ship	nofis-ship-s	nofis-spi	nofis-swr	nofis-swr-s	nogpu	nogpu-s	s	spi	spi-nw	spi-nw-1proc	spi-nw-nofis	spi-nw-nofis-s	spi-nw-s	spi-s	swr	swr-s	vt	vt-s	wmfme	wr-dc2-o	xorig	xorig-s
linux1804-64-shippable-qr/opt																																1
windows11-64-shippable-qr/opt																																1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2024-12-16&endday=2024-12-22&tree=all

Comment 27

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 28

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=490159683&repo=autoland

Comment 29

[Intermittent Failures Robot]

Summary

2 failures in 4508 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• linux1804-64-qr: 1
  • opt: 1
• macosx1470-64: 1
  • opt: 1

Table

	no_variant
linux1804-64-qr/-/opt	1
macosx1470-64/-/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-01-13&endday=2025-01-19&tree=all

Comment 30

[Intermittent Failures Robot]

Summary

1 failures in 4035 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-hw-a55-14-0-aarch64: 1
  • opt: 1

Table

	nofis
android-hw-a55-14-0-aarch64--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-01-20&endday=2025-01-26&tree=all

Comment 31

[Intermittent Failures Robot]

Summary

2 failures in 4309 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-qr: 1
  • opt: 1
• windows11-64-qr: 1
  • opt: 1

Table

	no_variant
linux1804-64-qr--/opt	1
windows11-64-qr--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-02-03&endday=2025-02-09&tree=all

Comment 32

[Intermittent Failures Robot]

Summary

1 failures in 4340 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • opt: 1

Table

	no_variant
linux1804-64-qr--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-02-17&endday=2025-02-23&tree=all

Comment 33

[Intermittent Failures Robot]

Summary

2 failures in 4564 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• android-5-0-armv7: 2
  • opt: 2

Table

	no_variant
android-5-0-armv7--/opt	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-02-24&endday=2025-03-02&tree=all

Comment 34

[Intermittent Failures Robot]

Summary

3 failures in 4009 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform and build breakdown:

• android-hw-a55-14-0-aarch64: 2
  • opt: 2
• linux1804-64-nightlyasrelease-qr: 1
  • opt: 1

Table

	no_variant	nofis
android-hw-a55-14-0-aarch64--/opt		2
linux1804-64-nightlyasrelease-qr--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-03-03&endday=2025-03-09&tree=all

Comment 35

[Serban Stanca [:SerbanS]]

Tier 1 link failure here.

Comment 36

[amarc]

Hi, there's been a recent spike in failures , could you please take a look? Thanks.

Comment 37

[Intermittent Failures Robot]

Summary

38 failures in 4826 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #49 most frequent failure this week.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 37
• autoland: 1

Platform and build breakdown:

• linux1804-64-nightlyasrelease-qr: 4
  • opt: 4
• linux1804-64-qr: 8
  • opt: 8
• macosx1470-64-nightlyasrelease: 5
  • opt: 5
• macosx1470-64: 5
  • opt: 5
• windows11-64-24h2-nightlyasrelease: 6
  • opt: 6
• windows11-64-24h2: 10
  • opt: 10

Table

	no_variant
linux1804-64-nightlyasrelease-qr--/opt	4
linux1804-64-qr--/opt	8
macosx1470-64-nightlyasrelease--/opt	5
macosx1470-64--/opt	5
windows11-64-24h2-nightlyasrelease--/opt	6
windows11-64-24h2--/opt	10

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-03-10&endday=2025-03-16&tree=all

Comment 38

[Andrej (:aglavic)]

Thanks, we're investigating!

Comment 39

[Andrej (:aglavic)]

Error seems to have not appeared in last week, I'm removing the needinfo for myself but going to continue work on this here: Bug 1804694

Comment 40

[Intermittent Failures Robot]

7 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7

Table

	no_variant	nofis
android-hw-a55-14-0-aarch64--/opt	2	3
windows11-64-24h2--/opt	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-03-31&endday=2025-04-06&tree=all

Comment 41

[Intermittent Failures Robot]

2 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Table

	no_variant	nofis
android-hw-a55-14-0-aarch64--/opt		1
macosx1400-64-qr--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-04-14&endday=2025-04-20&tree=all

Comment 42

[Intermittent Failures Robot]

1 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Table

	no_variant
macosx1470-64--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-04-21&endday=2025-04-27&tree=all

Comment 43

[Intermittent Failures Robot]

5 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 3

Table

	no_variant
macosx1400-64-qr--/opt	1
macosx1470-64-nightlyasrelease--/opt	2
macosx1470-64--/opt	1
macosx1500-aarch64--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-04-28&endday=2025-05-04&tree=all

Comment 44

[Intermittent Failures Robot]

1 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Table

	no_variant
windows11-x86_64/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-05-05&endday=2025-05-11&tree=all

Comment 45

[Intermittent Failures Robot]

1 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Table

	nofis
android-aarch64/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-05-12&endday=2025-05-18&tree=all

Comment 46

[Intermittent Failures Robot]

4 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 1

Table

	no_variant	nofis
android-aarch64/opt	3	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-06-02&endday=2025-06-08&tree=all

Comment 47

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 48

[Intermittent Failures Robot]

2 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Table

	nofis
android-aarch64/opt	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2025-07-14&endday=2025-07-20&tree=all

Comment 49

[Intermittent Failures Robot]

1 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Table

	no_variant
linux1804-x86_64/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1884354&startday=2026-03-30&endday=2026-04-05&tree=all