1576971 - browsertime toolchain task doesn't fail hard enough

Status: RESOLVED FIXED

(Testing :: Performance, defect)

Description

[Nick Alexander :nalexander [he/him] (parental leave until 2020!)]

Bug 1564256 added a toolchain-browsertime task. It can fail during download, like this task:

[task 2019-08-26T21:53:45.754Z] # We don't install ImageMagick, so this will fail.  Continue.
[task 2019-08-26T21:53:45.754Z] ./mach browsertime --setup || true
[task 2019-08-26T21:53:45.754Z] + ./mach browsertime --setup
[task 2019-08-26T21:53:46.385Z] 
[task 2019-08-26T21:53:46.386Z] Creating default state directory: /builds/worker/.mozbuild
[task 2019-08-26T21:53:46.386Z]  0:00.54 Downloading to temporary location /builds/worker/.mozbuild/cache/browsertime/d64f870ceed29d8a-ffmpeg-4.0.3-64bit-static.tar.xz
[task 2019-08-26T21:53:46.720Z]  0:00.87 Downloading... 0.0 %
[task 2019-08-26T21:53:46.721Z]  0:00.87 Downloading... 100.0 %
[task 2019-08-26T21:53:46.724Z] Error running mach:
[task 2019-08-26T21:53:46.724Z] 
[task 2019-08-26T21:53:46.724Z]     ['browsertime', '--setup']
[task 2019-08-26T21:53:46.724Z] 
[task 2019-08-26T21:53:46.724Z] The error occurred in code that was called by the mach command. This is either
[task 2019-08-26T21:53:46.724Z] a bug in the called code itself or in the way that mach is calling it.
[task 2019-08-26T21:53:46.724Z] You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2019-08-26T21:53:46.724Z] isn't, please use |./mach busted file| to report it. If |./mach busted| is
[task 2019-08-26T21:53:46.724Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2019-08-26T21:53:46.724Z] 
[task 2019-08-26T21:53:46.724Z] If filing a bug, please include the full output of mach, including this error
[task 2019-08-26T21:53:46.724Z] message.
[task 2019-08-26T21:53:46.724Z] 
[task 2019-08-26T21:53:46.724Z] The details of the failure are as follows:
[task 2019-08-26T21:53:46.724Z] 
[task 2019-08-26T21:53:46.724Z] HTTPError: 406 Client Error: Not Acceptable for url: https://www.johnvansickle.com/ffmpeg/old-releases/ffmpeg-4.0.3-64bit-static.tar.xz
[task 2019-08-26T21:53:46.724Z] 
[task 2019-08-26T21:53:46.724Z]   File "/builds/worker/workspace/build/src/tools/browsertime/mach_commands.py", line 399, in browsertime
[task 2019-08-26T21:53:46.724Z]     return self.setup(should_clobber=clobber)
[task 2019-08-26T21:53:46.724Z]   File "/builds/worker/workspace/build/src/tools/browsertime/mach_commands.py", line 197, in setup
[task 2019-08-26T21:53:46.724Z]     archive = artifact_cache.fetch(fetch['url'])
[task 2019-08-26T21:53:46.724Z]   File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/artifact_cache.py", line 208, in fetch
[task 2019-08-26T21:53:46.725Z]     dl.wait()
[task 2019-08-26T21:53:46.725Z]   File "/builds/worker/workspace/build/src/third_party/python/dlmanager/dlmanager/manager.py", line 101, in wait
[task 2019-08-26T21:53:46.725Z]     self.raise_if_error()
[task 2019-08-26T21:53:46.725Z]   File "/builds/worker/workspace/build/src/third_party/python/dlmanager/dlmanager/manager.py", line 116, in raise_if_error
[task 2019-08-26T21:53:46.725Z]     six.reraise(*self.__error)
[task 2019-08-26T21:53:46.725Z]   File "/builds/worker/workspace/build/src/third_party/python/dlmanager/dlmanager/manager.py", line 175, in _download
[task 2019-08-26T21:53:46.725Z]     response.raise_for_status()
[task 2019-08-26T21:53:46.725Z]   File "/builds/worker/workspace/build/src/third_party/python/requests/requests/models.py", line 840, in raise_for_status
[task 2019-08-26T21:53:46.725Z]     raise HTTPError(http_error_msg, response=self)
[task 2019-08-26T21:53:46.743Z] + true

That's not good. This ticket tracks not running --check rather than suppressing all errors.

Comment 1

[Nick Alexander :nalexander [he/him] (parental leave until 2020!)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=10acc88cc34a53f9c7ba70034b82ccbc84ad9376&selectedJob=263730168 looks good: red, 'cuz the URL is busted from AWS right now.

Comment 2

[Nick Alexander :nalexander [he/him] (parental leave until 2020!)]

Attachment: Bug 1576971 - Fail browsertime toolchain harder. r?barret

Comment 3

[Nick Alexander :nalexander [he/him] (parental leave until 2020!)]

Attachment: Bug 1576971 - Post: Use github.com for fetching browsertime binary dependencies. r?barret

In automation, we install ffmpeg as part of mach browsertime --setup in the browsertime toolchain task. Those tasks run on Linux
64 from within AWS, and most of the hosts we hit (intermittently) deny
AWS traffic. Let's just use github.com in automation (and locally),
for all platforms, which will agree with upcoming fetch tasks.

Depends on D43655

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4ba838d11fc4
Fail browsertime toolchain harder. r=barret
https://hg.mozilla.org/integration/autoland/rev/49c4394e81be
Post: Use github.com for fetching browsertime binary dependencies. r=barret

Comment 5

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/4ba838d11fc4
https://hg.mozilla.org/mozilla-central/rev/49c4394e81be