1802922 - [meta] look into failing safari tests

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Kash Shampur [:kshampur] ⌚EST]

Currently the following tests consistently fail in Safari:

    wasm godot
    wasm misc
    matrix-react-bench
    webaudio
    gmail
    imgur
    netflix
    yahoo-mail
    fandom
    speedometer3

A common failure message among many of the tests seem to be timeout errors, but increasing the time has not been found to help.

e.g. message:

[task 2022-11-24T05:11:19.054Z] 05:11:19     INFO -  raptor-browsertime Info: Navigating to url about:blank iteration 7
[task 2022-11-24T05:11:24.161Z] 05:11:24     INFO -  raptor-browsertime Info: Navigating to primary url:https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase
[task 2022-11-24T05:11:24.161Z] 05:11:24     INFO -  raptor-browsertime Info: Cycle 0, waiting for 10000 ms
[task 2022-11-24T05:11:34.282Z] 05:11:34     INFO -  raptor-browsertime Info: Cycle 0, starting the measure
[task 2022-11-24T05:11:34.282Z] 05:11:34     INFO -  raptor-browsertime Info: Testing url https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase iteration 7
[task 2022-11-24T05:13:01.750Z] 05:13:01    ERROR -  raptor-browsertime Error: Browsertime failed to run
[task 2022-11-24T05:13:01.771Z] 05:13:01     INFO -  raptor-browsertime Info:     at Object.throwDecodedError (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/selenium-webdriver/lib/error.js:522:15)
[task 2022-11-24T05:13:01.771Z] 05:13:01     INFO -  raptor-browsertime Info:     at parseHttpResponse (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/selenium-webdriver/lib/http.js:549:13)
[task 2022-11-24T05:13:01.771Z] 05:13:01     INFO -  raptor-browsertime Info:     at Executor.execute (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/selenium-webdriver/lib/http.js:475:28)
[task 2022-11-24T05:13:01.771Z] 05:13:01     INFO -  raptor-browsertime Info:     at runMicrotasks (<anonymous>)
[task 2022-11-24T05:13:01.772Z] 05:13:01     INFO -  raptor-browsertime Info:     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[task 2022-11-24T05:13:01.772Z] 05:13:01     INFO -  raptor-browsertime Info:     at async Driver.execute (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/selenium-webdriver/lib/webdriver.js:735:17)
[task 2022-11-24T05:13:01.773Z] 05:13:01     INFO -  raptor-browsertime Info:     at async SeleniumRunner.loadAndWait (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/browsertime/lib/core/seleniumRunner.js:241:7)
[task 2022-11-24T05:13:01.773Z] 05:13:01     INFO -  raptor-browsertime Info:     at async Measure.start (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/browsertime/lib/core/engine/command/measure.js:224:9)
[task 2022-11-24T05:13:01.774Z] 05:13:01     INFO -  raptor-browsertime Info:     at async module.exports (/opt/worker/tasks/task_166926536438709/build/tests/raptor/browsertime/browsertime_pageload.js:400:5)
[task 2022-11-24T05:13:01.774Z] 05:13:01     INFO -  raptor-browsertime Info:     at async /opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/browsertime/lib/core/engine/run.js:7:9
[task 2022-11-24T05:13:01.775Z] 05:13:01     INFO -  raptor-browsertime Info:     at async Iteration.run (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/browsertime/lib/core/engine/iteration.js:223:9)
[task 2022-11-24T05:13:01.775Z] 05:13:01     INFO -  raptor-browsertime Info:     at async Engine.runByScript (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/browsertime/lib/core/engine/index.js:266:20)
[task 2022-11-24T05:13:01.775Z] 05:13:01     INFO -  raptor-browsertime Info:     at async run (/opt/worker/tasks/task_166926536438709/fetches/browsertime/node_modules/browsertime/bin/browsertime.js:74:22)
[task 2022-11-24T05:13:01.775Z] 05:13:01 CRITICAL -  raptor-browsertime Critical: TimeoutError
[task 2022-11-24T05:13:01.962Z] 05:13:01    ERROR -  Traceback (most recent call last):
[task 2022-11-24T05:13:01.962Z] 05:13:01     INFO -    File "/opt/worker/tasks/task_166926536438709/build/tests/raptor/raptor/raptor.py", line 207, in <module>
[task 2022-11-24T05:13:01.962Z] 05:13:01     INFO -      main()
[task 2022-11-24T05:13:01.963Z] 05:13:01     INFO -    File "/opt/worker/tasks/task_166926536438709/build/tests/raptor/raptor/raptor.py", line 153, in main
[task 2022-11-24T05:13:01.963Z] 05:13:01     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2022-11-24T05:13:01.963Z] 05:13:01     INFO -    File "/opt/worker/tasks/task_166926536438709/build/tests/raptor/raptor/perftest.py", line 471, in run_tests
[task 2022-11-24T05:13:01.963Z] 05:13:01     INFO -      self.run_test(test, timeout=int(test.get("page_timeout")))
[task 2022-11-24T05:13:01.964Z] 05:13:01     INFO -    File "/opt/worker/tasks/task_166926536438709/build/tests/raptor/raptor/browsertime/base.py", line 803, in run_test
[task 2022-11-24T05:13:01.964Z] 05:13:01     INFO -      raise Exception(self.browsertime_failure)
[task 2022-11-24T05:13:01.964Z] 05:13:01     INFO -  Exception: TimeoutError

and

[task 2022-11-24T06:42:12.291Z] 06:42:12     INFO -  raptor-browsertime Info: Cycle 0, waiting for 1000 ms
[task 2022-11-24T06:42:13.312Z] 06:42:13     INFO -  raptor-browsertime Info: Cycle 0, starting the measure
[task 2022-11-24T06:42:13.313Z] 06:42:13     INFO -  raptor-browsertime Info: Testing url http://localhost:49266/wasm-godot/index.html iteration 1
[task 2022-11-24T06:42:18.679Z] 06:42:18     INFO -  raptor-browsertime Info: Benchmark custom metric collection
[task 2022-11-24T06:42:18.685Z] 06:42:18     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:21.706Z] 06:42:21     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:24.717Z] 06:42:24     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:27.738Z] 06:42:27     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:30.749Z] 06:42:30     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:33.831Z] 06:42:33     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:36.889Z] 06:42:36     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:39.951Z] 06:42:39     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:42.987Z] 06:42:42     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:46.064Z] 06:42:46     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:49.110Z] 06:42:49     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:52.132Z] 06:42:52     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:55.208Z] 06:42:55     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:42:58.227Z] 06:42:58     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:01.301Z] 06:43:01     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:04.343Z] 06:43:04     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:07.427Z] 06:43:07     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:10.511Z] 06:43:10     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:13.586Z] 06:43:13     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:16.648Z] 06:43:16     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:19.653Z] 06:43:19     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:22.686Z] 06:43:22     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:25.714Z] 06:43:25     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:28.794Z] 06:43:28     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:31.869Z] 06:43:31     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:34.946Z] 06:43:34     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:37.984Z] 06:43:37     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:41.008Z] 06:43:41     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:44.051Z] 06:43:44     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:47.133Z] 06:43:47     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:50.192Z] 06:43:50     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:53.232Z] 06:43:53     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:56.310Z] 06:43:56     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:43:59.388Z] 06:43:59     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:44:02.469Z] 06:44:02     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:44:05.535Z] 06:44:05     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:44:08.603Z] 06:44:08     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:44:11.639Z] 06:44:11     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:44:14.705Z] 06:44:14     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:44:17.715Z] 06:44:17     INFO -  raptor-browsertime Info: Waiting 3000 ms for data from benchmark...
[task 2022-11-24T06:44:20.754Z] 06:44:20    ERROR -  raptor-browsertime Error: Browsertime failed to run
[task 2022-11-24T06:44:20.775Z] 06:44:20     INFO -  raptor-browsertime Info: Navigating to about:blank
[task 2022-11-24T06:44:20.775Z] 06:44:20     INFO -  raptor-browsertime Info: Navigating to url about:blank iteration 1
[task 2022-11-24T06:44:20.775Z] 06:44:20 CRITICAL -  raptor-browsertime Critical: Benchmark timed out. Aborting...
[task 2022-11-24T06:44:20.777Z] 06:44:20    ERROR -  Traceback (most recent call last):
[task 2022-11-24T06:44:20.778Z] 06:44:20     INFO -    File "/opt/worker/tasks/task_166927201543383/build/tests/raptor/raptor/raptor.py", line 207, in <module>
[task 2022-11-24T06:44:20.778Z] 06:44:20     INFO -      main()
[task 2022-11-24T06:44:20.778Z] 06:44:20     INFO -    File "/opt/worker/tasks/task_166927201543383/build/tests/raptor/raptor/raptor.py", line 153, in main
[task 2022-11-24T06:44:20.778Z] 06:44:20     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2022-11-24T06:44:20.779Z] 06:44:20     INFO -    File "/opt/worker/tasks/task_166927201543383/build/tests/raptor/raptor/perftest.py", line 471, in run_tests
[task 2022-11-24T06:44:20.779Z] 06:44:20     INFO -      self.run_test(test, timeout=int(test.get("page_timeout")))
[task 2022-11-24T06:44:20.779Z] 06:44:20     INFO -    File "/opt/worker/tasks/task_166927201543383/build/tests/raptor/raptor/browsertime/base.py", line 803, in run_test
[task 2022-11-24T06:44:20.780Z] 06:44:20     INFO -      raise Exception(self.browsertime_failure)
[task 2022-11-24T06:44:20.780Z] 06:44:20     INFO -  Exception: Benchmark timed out. Aborting...
[task 2022-11-24T06:44:20.955Z] 06:44:20    ERROR - Return code: 1

Worth mentioning some benchmarks do work locally with the repeating Waiting 3000 ms message, eventually continuing and passing. But fail on CI despite increasing time out.

This bug is meta tracker for now, as each test might need a different solution

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The matrix-react benchmark issue is logged here: https://github.com/mozilla-spidermonkey/matrix-react-bench/issues/4

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

quick update

https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=beb35d7d61a2f3e9f2738f9f068ffc6e0cd8d7b8&selectedTaskRun=I708DOEVS-6pTHTVPKjbIg.0

here is a recent push on a pool of machines with Safari upgraded that relops was working on

comparing the previous failures to now (i've re-enabled previously disabled tests here), i think overall it seems the Safari upgrade has more passes than fails (more important in particular, is SP3 passing!)

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

Plan is for Safari upgrades to be rolled out next Monday by relops
https://mozilla-hub.atlassian.net/browse/RELOPS-381?focusedCommentId=712583

afterwhich we can enable/disable the tests as needed based on what was found in my previous comment

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

safari upgrades have been rolled out in Bug 1812409
I will work on a patch that disables/re-enables the new set of passing/failing tests

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1802922 - Finalize list of Safari performance tests for OSX 10.15. r?#perftest

Previously the OSX machines had a safari version (14.x.x) that was not
up to date (15.x.x) for the 10.15.7 machines. With this recent safari
version upgrade, the previous test set needs to be updated due since now
some tests that used to permafail no longer do.

Since we can't upgrade safari any further on these machines, this is as
best as we currently can do. The test set will be revisited in tp7 work
and more generally speaking, the apple silicon machines will have the
more up to date Safari version soon.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c3b23fbbe898
Finalize list of Safari performance tests for OSX 10.15. r=perftest-reviewers,sparky

Comment 7

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/c3b23fbbe898