Investigate failures of browsertime speedometer test on Windows
Categories
(Testing :: Raptor, defect, P1)
Tracking
(firefox72 fixed)
| Tracking | Status | |
|---|---|---|
| firefox72 | --- | fixed |
People
(Reporter: Bebe, Assigned: Bebe)
References
Details
Attachments
(1 file)
browsertime speedometer test on Windows is perma failing.
Investigate and fix the failures.
|
Assignee | |
Updated•5 years ago
|
|
||
Updated•5 years ago
|
|
|
||
Updated•5 years ago
|
| Comment hidden (Intermittent Failures Robot) |
|
|
Assignee | |
Comment 2•5 years ago
|
||
After some effort managed to get a windows env setup.
Ran the test and it run without any issues on Windows.
Investigating the try failure:
Try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1c1ace7e196799c24d8146ece37a7108f8918954
|
|
Assignee | |
Comment 3•5 years ago
•
|
||
the failure is:
[task 2019-11-20T16:47:44.856Z] 16:47:44 ERROR - Traceback (most recent call last):
[task 2019-11-20T16:47:44.856Z] 16:47:44 INFO - File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 1808, in <module>
[task 2019-11-20T16:47:44.856Z] 16:47:44 INFO - main()
[task 2019-11-20T16:47:44.856Z] 16:47:44 INFO - File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 1776, in main
[task 2019-11-20T16:47:44.856Z] 16:47:44 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 272, in run_tests
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - self.run_test(test, timeout=int(test.get('page_timeout')))
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 669, in run_test
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - self.run_test_setup(test)
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 625, in run_test_setup
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - self.benchmark = Benchmark(self.config, test)
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - File "C:\Users\task_1574265915\build\tests\raptor\raptor\benchmark.py", line 48, in __init__
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - shutil.copytree(path, os.path.join(self.bench_dir, name))
[task 2019-11-20T16:47:44.857Z] 16:47:44 INFO - File "c:\mozilla-build\python\Lib\shutil.py", line 222, in copytree
[task 2019-11-20T16:47:44.858Z] 16:47:44 INFO - raise Error, errors
[task 2019-11-20T16:47:44.861Z] 16:47:44 INFO - shutil.Error: [( [('C:/Users/task_1574265915/fetches\\browsertime\\node_modules\\.bin\\browsertime', 'C:\\Users\\task_1574265915\\build\\tests\\raptor\\raptor\\tests\\webkit\\PerformanceTests\\browsertime\\node_modules\\.bin\\browsertime', "[Errno 22] invalid mode ('rb') or filename: 'C:/Users/task_1574265915/fetches\\\\browsertime\\\\node_modules\\\\.bin\\\\browsertime'")
which is triggered in https://searchfox.org/mozilla-central/source/testing/raptor/raptor/benchmark.py#39
Do we need to add special artifacts for benchmark tests that is not added to raptor browser time for windows?
|
|
||
Comment 4•5 years ago
|
||
Sorry :bebe I really don't know the answer to that off-hand. I wouldn't think so...
|
|
Assignee | |
Comment 5•5 years ago
|
||
the failure happens when we try to shutil.copytree the content of MOZ_FETCHES_DIR to a bench_dir path.
In the case of windows build there are multiple failures when copying the dir (permissions name to long etc.)
all because in case of a browsertime build that folder contains: ['browsertime', 'ffmpeg-4.1.1-win64-static', 'geckodriver.exe', 'node']
Do we use fetch tasks to store benchmark resources? If not maybe we can avoid this procedure.
Otherwise we need to hardcore and avoid the browsertime resources from copy
|
|
Assignee | |
Comment 6•5 years ago
|
||
|
|
||
Comment 7•5 years ago
|
||
(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #5)
the failure happens when we try to shutil.copytree the content of MOZ_FETCHES_DIR to a bench_dir path.
In the case of windows build there are multiple failures when copying the dir (permissions name to long etc.)
all because in case of a browsertime build that folder contains: ['browsertime', 'ffmpeg-4.1.1-win64-static', 'geckodriver.exe', 'node']Do we use fetch tasks to store benchmark resources? If not maybe we can avoid this procedure.
Otherwise we need to hardcore and avoid the browsertime resources from copy
Hey :bebe, yes some benchmarks don't have their source in-tree so they use fetch tasks. You can see in the taskcluster config which benchmarks use fetch tasks, i.e. [0].
Speedometer doesn't use a fetch task, because all it's source is already in-tree [1].
[0] https://searchfox.org/mozilla-central/rev/652014ca1183c56bc5f04daf01af180d4e50a91c/taskcluster/ci/test/raptor.yml#399
[1] https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/Speedometer
|
|
Assignee | |
Comment 8•5 years ago
|
||
Pushed by fstrugariu@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/24fc1c1ae789 Investigate failures of browsertime speedometer test on Windows r=perftest-reviewers,rwood
|
||
Comment 10•5 years ago
|
||
| bugherder | ||
Description
•