1807754 - Make matrix-react-bench pass on Safari

Status: RESOLVED WONTFIX

(Testing :: Raptor, defect, P3)

Description

[Kash Shampur [:kshampur] ⌚EST]

on CI and locally, you will see a timeout error that looks something like this

22:24:48     INFO -  raptor-browsertime Info: Navigating to url about:blank iteration 1
22:24:53     INFO -  raptor-browsertime Info: Cycle 0, waiting for 1000 ms
22:24:54     INFO -  raptor-browsertime Info: Cycle 0, starting the measure
22:24:54     INFO -  raptor-browsertime Info: Testing url http://127.0.0.1:54245/matrix-react-bench/matrix_demo.html iteration 1
22:24:59    ERROR -  raptor-browsertime Error: Browsertime failed to run
22:24:59    ERROR -  raptor-browsertime Error: Browsertime failed to run
22:25:00 CRITICAL -  raptor-browsertime Critical: Failed to execute user script: TimeoutError
22:25:00    ERROR -  Traceback (most recent call last):
22:25:00     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/testing/raptor/raptor/raptor.py", line 206, in <module>
22:25:00     INFO -      main()
22:25:00     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/testing/raptor/raptor/raptor.py", line 152, in main
22:25:00     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
22:25:00     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/testing/raptor/raptor/perftest.py", line 471, in run_tests
22:25:00     INFO -      self.run_test(test, timeout=int(test.get("page_timeout")))
22:25:00     INFO -    File "/Users/kshampur/wrkspc/mozilla-unified/testing/raptor/raptor/browsertime/base.py", line 816, in run_test
22:25:00     INFO -      raise Exception(self.browsertime_failure)
22:25:00     INFO -  Exception: Failed to execute user script: TimeoutError

additionally there is a bug filed here with more details https://github.com/mozilla-spidermonkey/matrix-react-bench/issues/4

having MRB passing on safari will be useful for some upcoming benchmark work

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

adding some debugging statements in the raptor line_handler, looks like cpuBenchmark is failing resulting in the timeout error we see
Error: Couldn't execute script named cpuBenchmark error:TimeoutError

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

Okay so looks like it times out on safari because of this variable being too large in cpuBenchmark.js?
https://github.com/sitespeedio/browsertime/blob/main/browserscripts/browser/cpuBenchmark.js#L5

I verified this by manually knocking off a couple zeros locally, and it stopped timing out and is running. (I wonder if this is the reason for the other benchmark timeout failures)

Not sure why this affects Safari matrix-react-bench, but not some of the other passing benchmarks.

So I plan to make a patch that adds a safari+mrb exception in _kill_browsertime_process()
https://searchfox.org/mozilla-central/source/testing/raptor/raptor/browsertime/base.py#622

(this func was originally made to handle such exceptions during the line handling anyway e.g. the current xpath 2fa thing for login logic that is present there)

edit
never mind, that approach doesn't quite work, as ignoring the timeouterror from cpubenchmark in the proposed way still has seleniumrunner attempt to add undefined data to the browser which creates an error..

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

removing myself as assignee for the time being.
I will be reprioritizing Bug 1809535 once Safari upgrades are done