1593598 - Investigate failures of browsertime speedometer test on Windows

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Florin Strugariu [:Bebe]]

browsertime speedometer test on Windows is perma failing.

Investigate and fix the failures.

Comment 1

[Intermittent Failures Robot]

2 failures in 4850 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows10-64-qr: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1593598&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 2

[Florin Strugariu [:Bebe]]

After some effort managed to get a windows env setup.
Ran the test and it run without any issues on Windows.

Investigating the try failure:

Try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1c1ace7e196799c24d8146ece37a7108f8918954

Comment 3

[Florin Strugariu [:Bebe]]

the failure is:

[task 2019-11-20T16:47:44.856Z] 16:47:44    ERROR -  Traceback (most recent call last):
[task 2019-11-20T16:47:44.856Z] 16:47:44     INFO -    File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 1808, in <module>
[task 2019-11-20T16:47:44.856Z] 16:47:44     INFO -      main()
[task 2019-11-20T16:47:44.856Z] 16:47:44     INFO -    File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 1776, in main
[task 2019-11-20T16:47:44.856Z] 16:47:44     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -    File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 272, in run_tests
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -      self.run_test(test, timeout=int(test.get('page_timeout')))
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -    File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 669, in run_test
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -      self.run_test_setup(test)
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -    File "C:\Users\task_1574265915\build\tests\raptor\raptor\raptor.py", line 625, in run_test_setup
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -      self.benchmark = Benchmark(self.config, test)
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -    File "C:\Users\task_1574265915\build\tests\raptor\raptor\benchmark.py", line 48, in __init__
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -      shutil.copytree(path, os.path.join(self.bench_dir, name))
[task 2019-11-20T16:47:44.857Z] 16:47:44     INFO -    File "c:\mozilla-build\python\Lib\shutil.py", line 222, in copytree
[task 2019-11-20T16:47:44.858Z] 16:47:44     INFO -      raise Error, errors
[task 2019-11-20T16:47:44.861Z] 16:47:44     INFO -  shutil.Error: [( [('C:/Users/task_1574265915/fetches\\browsertime\\node_modules\\.bin\\browsertime', 'C:\\Users\\task_1574265915\\build\\tests\\raptor\\raptor\\tests\\webkit\\PerformanceTests\\browsertime\\node_modules\\.bin\\browsertime', "[Errno 22] invalid mode ('rb') or filename: 'C:/Users/task_1574265915/fetches\\\\browsertime\\\\node_modules\\\\.bin\\\\browsertime'")

which is triggered in https://searchfox.org/mozilla-central/source/testing/raptor/raptor/benchmark.py#39

Do we need to add special artifacts for benchmark tests that is not added to raptor browser time for windows?

Comment 4

[Robert Wood [:rwood]]

Sorry :bebe I really don't know the answer to that off-hand. I wouldn't think so...

Comment 5

[Florin Strugariu [:Bebe]]

the failure happens when we try to shutil.copytree the content of MOZ_FETCHES_DIR to a bench_dir path.

In the case of windows build there are multiple failures when copying the dir (permissions name to long etc.)
all because in case of a browsertime build that folder contains: ['browsertime', 'ffmpeg-4.1.1-win64-static', 'geckodriver.exe', 'node']

Do we use fetch tasks to store benchmark resources? If not maybe we can avoid this procedure.
Otherwise we need to hardcore and avoid the browsertime resources from copy

Comment 6

[Florin Strugariu [:Bebe]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ebb6d5bced429fcffc4245ee4aa14ffc0f2479b9

Comment 7

[Robert Wood [:rwood]]

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #5)

the failure happens when we try to shutil.copytree the content of MOZ_FETCHES_DIR to a bench_dir path.

In the case of windows build there are multiple failures when copying the dir (permissions name to long etc.)
all because in case of a browsertime build that folder contains: ['browsertime', 'ffmpeg-4.1.1-win64-static', 'geckodriver.exe', 'node']

Do we use fetch tasks to store benchmark resources? If not maybe we can avoid this procedure.
Otherwise we need to hardcore and avoid the browsertime resources from copy

Hey :bebe, yes some benchmarks don't have their source in-tree so they use fetch tasks. You can see in the taskcluster config which benchmarks use fetch tasks, i.e. [0].

Speedometer doesn't use a fetch task, because all it's source is already in-tree [1].

[0] https://searchfox.org/mozilla-central/rev/652014ca1183c56bc5f04daf01af180d4e50a91c/taskcluster/ci/test/raptor.yml#399
[1] https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/Speedometer

Comment 8

[Florin Strugariu [:Bebe]]

Attachment: Bug 1593598 - Investigate failures of browsertime speedometer test on Windows

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/24fc1c1ae789
Investigate failures of browsertime speedometer test on Windows r=perftest-reviewers,rwood

Comment 10

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/24fc1c1ae789