1890740 - Interminttent Btime-car [tier 2] raptor-browsertime Critical: [custom-car] Could not run JavaScript return !(this.benchmarkClient._isRunning)

Status: REOPENED

(Testing :: Raptor, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=454151521&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Jo70E5R4ScKplWIywXNsug/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-04-10T10:15:23.351Z] 10:15:23     INFO -  raptor-browsertime Info: Testing url http://127.0.0.1:49269/index.html iteration 1
[task 2024-04-10T10:15:31.876Z] 10:15:31     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from speedometer...
[task 2024-04-10T10:15:35.020Z] 10:15:35     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from speedometer...
[task 2024-04-10T10:15:38.103Z] 10:15:38     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from speedometer...
[task 2024-04-10T10:15:41.179Z] 10:15:41     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from speedometer...
[task 2024-04-10T10:15:44.286Z] 10:15:44     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from speedometer...
[task 2024-04-10T10:15:47.400Z] 10:15:47     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from speedometer...
[task 2024-04-10T10:15:50.434Z] 10:15:50     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from speedometer...
[task 2024-04-10T10:15:53.646Z] 10:15:53     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from speedometer...
[task 2024-04-10T10:15:56.737Z] 10:15:56    ERROR -  raptor-browsertime Error: [custom-car] Browsertime failed to run
[task 2024-04-10T10:15:56.737Z] 10:15:56     INFO -  raptor-browsertime Info: from tab crashed
[task 2024-04-10T10:15:56.737Z] 10:15:56     INFO -  raptor-browsertime Info:   (Session info: chrome=125.0.6410.0)
[task 2024-04-10T10:15:56.737Z] 10:15:56    ERROR -  raptor-browsertime Error: [custom-car] Browsertime failed to run
[task 2024-04-10T10:15:56.843Z] 10:15:56     INFO -  TEST-INFO | screencapture: exit 0
[task 2024-04-10T10:15:56.843Z] 10:15:56 CRITICAL -  raptor-browsertime Critical: [custom-car] Could not run JavaScript return !(this.benchmarkClient._isRunning)
[task 2024-04-10T10:15:57.292Z] 10:15:57    ERROR -  Traceback (most recent call last):
[task 2024-04-10T10:15:57.292Z] 10:15:57     INFO -    File "/opt/worker/tasks/task_171274398212200/build/tests/raptor/raptor/raptor.py", line 188, in <module>
[task 2024-04-10T10:15:57.292Z] 10:15:57     INFO -      main()
[task 2024-04-10T10:15:57.293Z] 10:15:57     INFO -    File "/opt/worker/tasks/task_171274398212200/build/tests/raptor/raptor/raptor.py", line 137, in main
[task 2024-04-10T10:15:57.293Z] 10:15:57     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2024-04-10T10:15:57.293Z] 10:15:57     INFO -                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2024-04-10T10:15:57.293Z] 10:15:57     INFO -    File "/opt/worker/tasks/task_171274398212200/build/tests/raptor/raptor/perftest.py", line 499, in run_tests
[task 2024-04-10T10:15:57.293Z] 10:15:57     INFO -      self.run_test(test, timeout=int(test.get("page_timeout")))
[task 2024-04-10T10:15:57.294Z] 10:15:57     INFO -    File "/opt/worker/tasks/task_171274398212200/build/tests/raptor/raptor/browsertime/base.py", line 1051, in run_test
[task 2024-04-10T10:15:57.294Z] 10:15:57     INFO -      raise Exception(self.browsertime_failure)
[task 2024-04-10T10:15:57.294Z] 10:15:57     INFO -  Exception: Could not run JavaScript return !(this.benchmarkClient._isRunning)
[task 2024-04-10T10:15:57.333Z] 10:15:57     INFO - Return code: 1
[task 2024-04-10T10:15:57.333Z] 10:15:57  WARNING - setting return code to 1
[task 2024-04-10T10:15:57.333Z] 10:15:57     INFO - Copying Raptor results to upload dir:
[task 2024-04-10T10:15:57.333Z] 10:15:57     INFO - /opt/worker/tasks/task_171274398212200/build/blobber_upload_dir/perfherder-data.json
[task 2024-04-10T10:15:57.333Z] 10:15:57     INFO - Copying raptor results from /opt/worker/tasks/task_171274398212200/build/raptor.json to /opt/worker/tasks/task_171274398212200/build/blobber_upload_dir/perfherder-data.json
[task 2024-04-10T10:15:57.334Z] 10:15:57 CRITICAL - Error copying results /opt/worker/tasks/task_171274398212200/build/raptor.json to upload dir /opt/worker/tasks/task_171274398212200/build/blobber_upload_dir/perfherder-data.json
[task 2024-04-10T10:15:57.334Z] 10:15:57     INFO - [Errno 2] No such file or directory: '/opt/worker/tasks/task_171274398212200/build/raptor.json'
[task 2024-04-10T10:15:57.334Z] 10:15:57     INFO - Running post-action listener: _package_coverage_data
[task 2024-04-10T10:15:57.334Z] 10:15:57     INFO - Running post-action listener: _resource_record_post_action
[task 2024-04-10T10:15:57.334Z] 10:15:57     INFO - Running post-action listener: process_java_coverage_data
[task 2024-04-10T10:15:57.334Z] 10:15:57     INFO - Running post-action listener: stop_device
[task 2024-04-10T10:15:57.334Z] 10:15:57     INFO - [mozharness: 2024-04-10 10:15:57.334858Z] Finished run-tests step (success)
[task 2024-04-10T10:15:57.334Z] 10:15:57     INFO - Running post-run listener: _resource_record_post_run
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - Total resource usage - Wall time: 79s; CPU: 8%; Read bytes: 137924608; Write bytes: 208691200; Read time: 1118; Write time: 997
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - TinderboxPrint: CPU usage<br/>8.4%
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - TinderboxPrint: I/O read bytes / time<br/>137,924,608 / 1,118
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - TinderboxPrint: I/O write bytes / time<br/>208,691,200 / 997
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - TinderboxPrint: CPU idle<br/>867.3 (91.7%)
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - TinderboxPrint: CPU system<br/>20.0 (2.1%)
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - TinderboxPrint: CPU user<br/>58.1 (6.1%)
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - TinderboxPrint: Swap in / out<br/>303,931,392 / 0
[task 2024-04-10T10:15:57.433Z] 10:15:57     INFO - install-chromium-distribution - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-04-10T10:15:57.438Z] 10:15:57     INFO - run-tests - Wall time: 79s; CPU: 8%; Read bytes: 137924608; Write bytes: 208691200; Read time: 1118; Write time: 997
[task 2024-04-10T10:15:57.594Z] 10:15:57  WARNING - returning nonzero exit status 1
[taskcluster 2024-04-10T10:15:57.643Z]    Exit Code: 1
[taskcluster 2024-04-10T10:15:57.643Z]    User Time: 59.655937s
[taskcluster 2024-04-10T10:15:57.643Z]  Kernel Time: 13.850965s
[taskcluster 2024-04-10T10:15:57.643Z]    Wall Time: 2m28.375761s
[taskcluster 2024-04-10T10:15:57.643Z]       Result: FAILED
[taskcluster 2024-04-10T10:15:57.643Z] === Task Finished ===
[taskcluster 2024-04-10T10:15:57.643Z] Task Duration: 2m28.380378s
[taskcluster 2024-04-10T10:15:57.824Z] Uploading artifact public/logs/localconfig.json from file /opt/worker/tasks/task_171274398212200/logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-09T01:03:58.799Z
[taskcluster 2024-04-10T10:15:58.130Z] Uploading artifact public/test_info/mozilla-test-fail-screenshot_gfo97wwr.png from file /opt/worker/tasks/task_171274398212200/build/blobber_upload_dir/mozilla-test-fail-screenshot_gfo97wwr.png with content encoding "identity", mime type "image/png" and expiry 2024-07-09T01:03:58.799Z
[taskcluster 2024-04-10T10:15:58.379Z] Uploading artifact public/test_info/profile_resource-usage.json from file /opt/worker/tasks/task_171274398212200/build/blobber_upload_dir/profile_resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-09T01:03:58.799Z
[taskcluster 2024-04-10T10:15:58.636Z] Uploading artifact public/test_info/resource-usage.json from file /opt/worker/tasks/task_171274398212200/build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-07-09T01:03:58.799Z
[taskcluster 2024-04-10T10:15:58.921Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-07-09T01:03:58.799Z
[taskcluster:error] exit status 1

Comment 1

[Iulian Moraru]

Hi Kash! Can you also please take a look at this?
Thank you!

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

It is odd we went from complete failures in all CaR raptor tests to just sp3
I have this Try here where sp3 is passing https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=GQwCXBQJRPSrK6IfHSCcIg.0&tier=1%2C2%2C3&revision=d278d820b54c1c494e2acc037064ce605c68bcc0
(as part of the patch for Bug 1890666) hopefully that helps...

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

I triggered the chromium fetch and then retriggered the sp3 tasks and they seem to pass https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=chromium&revision=34fef7e7706b912024aa3ba8826647e5ddc0a0ab

more recent one https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=chromium&revision=e2456ba9a2f6f928e7e40cb84b59914fa989bf85

Comment 4

[Intermittent Failures Robot]

9 failures in 3235 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 9

Platform and build breakdown:

• macosx1015-64-shippable-qr: 7
  • opt: 7
• linux1804-64-shippable-qr: 1
  • opt: 1
• macosx1400-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-04-08&endday=2024-04-14&tree=all

Comment 5

[Intermittent Failures Robot]

1 failures in 4059 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1400-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-04-29&endday=2024-05-05&tree=all

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 7

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=460124139&repo=mozilla-central

Comment 8

[Intermittent Failures Robot]

3 failures in 4202 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• macosx1400-64-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-05-27&endday=2024-06-02&tree=all

Comment 9

[Kash Shampur [:kshampur] ⌚EST]

I've added some retriggers to the failing 1400and the pass. So this might be highly intermittent rather than perma.
Could be related to bug 1866703

Comment 10

[Intermittent Failures Robot]

2 failures in 3649 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• macosx1400-64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-06-03&endday=2024-06-09&tree=all

Comment 11

[Intermittent Failures Robot]

3 failures in 4625 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• macosx1400-64-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-06-10&endday=2024-06-16&tree=all

Comment 12

[Intermittent Failures Robot]

2 failures in 4116 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• android-hw-a51-11-0-aarch64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 13

[Intermittent Failures Robot]

1 failures in 3799 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1400-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-07-15&endday=2024-07-21&tree=all

Comment 14

[Intermittent Failures Robot]

1 failures in 4124 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1400-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-07-22&endday=2024-07-28&tree=all

Comment 15

[Intermittent Failures Robot]

1 failures in 3459 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1400-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-08-05&endday=2024-08-11&tree=all

Comment 16

[Intermittent Failures Robot]

1 failures in 2154 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1400-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-08-12&endday=2024-08-18&tree=all

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 18

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=487566633&repo=mozilla-central

Comment 19

[Intermittent Failures Robot]

Summary

1 failures in 3774 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1400-64-shippable-qr: 1
  • opt: 1

Table

	1proc	1proc-s	1proc-swr	a11y-checks	a11y-checks-s	a11y-checks-swr	a11y-checks-swr-s	condprof-s	fis	fis-hv	fis-hv-s	fis-s	headless	headless-s	http2	http3	mda-gpu	mda-gpu-s	msix	msix-s	no_variant	nofis	nofis-aab	nofis-s	nofis-ship	nofis-ship-s	nofis-spi	nofis-swr	nofis-swr-s	nogpu	nogpu-s	s	spi	spi-nw	spi-nw-1proc	spi-nw-nofis	spi-nw-nofis-s	spi-nw-s	spi-s	swr	swr-s	vt	vt-s	wmfme	wr-dc1-p	wr-dc2-o	wr-dc3-c	wr-dc3-c-s	xorig	xorig-s
macosx1400-64-shippable-qr/opt																																1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2024-12-16&endday=2024-12-22&tree=all

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 21

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=492203400&repo=mozilla-central

Comment 22

[Intermittent Failures Robot]

Summary

2 failures in 4566 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• windows10-64-qr: 2
  • opt: 2

Table

	no_variant
windows10-64-qr--/opt	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2025-01-27&endday=2025-02-02&tree=all

Comment 23

[Intermittent Failures Robot]

Summary

2 failures in 4309 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• macosx1400-64-qr: 1
  • opt: 1
• windows11-64-2009-hw-ref: 1
  • opt: 1

Table

	no_variant
macosx1400-64-qr--/opt	1
windows11-64-2009-hw-ref--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2025-02-03&endday=2025-02-09&tree=all

Comment 24

[Intermittent Failures Robot]

Summary

1 failures in 5022 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-hw-ref: 1
  • opt: 1

Table

	no_variant
windows11-64-2009-hw-ref--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2025-02-10&endday=2025-02-16&tree=all

Comment 25

[Intermittent Failures Robot]

Summary

3 failures in 4340 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• macosx1400-64-qr: 3
  • opt: 3

Table

	no_variant
macosx1400-64-qr--/opt	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1890740&startday=2025-02-17&endday=2025-02-23&tree=all