1594212 - vismet tasks are failing on mozilla-central

Status: RESOLVED WORKSFORME

(Testing :: Raptor, defect, P3)

Description

[Greg Mierzwinski [:sparky]]

This bug is to fix the failures of the visual-metrics task on mozilla-central, see here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedJob=274659097&tier=1%2C2%2C3&revision=2e1945e590ff9eb0907c6ee475efc31bdef5b207

Failure log:

[taskcluster 2019-11-05 17:08:58.259Z] Task ID: F5jeOLv2SByQfFFmq6pI9Q
[taskcluster 2019-11-05 17:08:58.259Z] Worker ID: i-0a5d9b3d223304c64
[taskcluster 2019-11-05 17:08:58.259Z] Worker Group: aws
[taskcluster 2019-11-05 17:08:58.259Z] Worker Node Type: c3.xlarge
[taskcluster 2019-11-05 17:08:58.259Z] Worker Type: t-linux-xlarge
[taskcluster 2019-11-05 17:08:58.259Z] Public IP: 54.187.232.167
[taskcluster 2019-11-05 17:08:58.259Z] Hostname: ip-10-144-24-183

[taskcluster 2019-11-05 17:08:59.423Z] Downloaded image is corrupted: (HTTP code 404) no such image - no such image: d6a4b322f45c28ed76cebd453d2b0708: No such image: d6a4b322f45c28ed76cebd453d2b0708:latest 
[taskcluster 2019-11-05 17:08:59.426Z] Downloading artifact "public/image.tar.zst" from task ID: eUwql9x9SryGRr7vhA0pQw.
[taskcluster 2019-11-05 17:09:03.228Z] Downloaded artifact successfully.
[taskcluster 2019-11-05 17:09:03.228Z] Downloaded 170.160 mb
[taskcluster 2019-11-05 17:09:03.228Z] Decompressing downloaded image
[taskcluster 2019-11-05 17:09:04.677Z] Loading docker image from downloaded archive.
[taskcluster 2019-11-05 17:09:16.871Z] Image 'public/image.tar.zst' from task 'eUwql9x9SryGRr7vhA0pQw' loaded.  Using image ID sha256:32d5f0bec57e33dd602e94b3ad01d07525a5ec792b83b7a69b6bc36799590052.
[taskcluster 2019-11-05 17:09:17.331Z] === Task Starting ===
[setup 2019-11-05T17:09:17.789Z] run-task started in /builds/worker
[volume 2019-11-05T17:09:17.790Z] changing ownership of volume /builds/worker/artifacts/ to 1000:1000
[volume 2019-11-05T17:09:17.790Z] changing ownership of volume /builds/worker/workspace/ to 1000:1000
[setup 2019-11-05T17:09:17.791Z] running as worker:worker
[setup 2019-11-05T17:09:17.791Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2019-11-05T17:09:17.791Z] fetching artifacts
[fetches 2019-11-05T17:09:17.791Z] executing ['/usr/bin/python3', '-u', '/builds/worker/bin/fetch-content', 'task-artifacts']
[fetches 2019-11-05T17:09:17.896Z] attempt 1/5
[fetches 2019-11-05T17:09:17.896Z] attempt 1/5
[fetches 2019-11-05T17:09:17.896Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json to /builds/worker/fetches/jobs.json
[fetches 2019-11-05T17:09:17.896Z] Downloading https://queue.taskcluster.net/v1/task/VrA04CmgSKGzVKR_iqEzLw/artifacts/public/visualmetrics.py to /builds/worker/fetches/visualmetrics.py
[fetches 2019-11-05T17:09:17.898Z] Downloading https://queue.taskcluster.net/v1/task/VrA04CmgSKGzVKR_iqEzLw/artifacts/public/visualmetrics.py
[fetches 2019-11-05T17:09:17.898Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json
[fetches 2019-11-05T17:09:18.439Z] Download failed: HTTP Error 404: Not Found
[fetches 2019-11-05T17:09:18.440Z] sleeping for 61.00s (attempt 1/5)
[fetches 2019-11-05T17:09:18.681Z] https://queue.taskcluster.net/v1/task/VrA04CmgSKGzVKR_iqEzLw/artifacts/public/visualmetrics.py resolved to 82756 bytes with sha256 9e587fb43c46dd0c37a15bc7688cf061c82592196eafe3ddd54e2ead27997d66 in 0.785s
[fetches 2019-11-05T17:10:19.446Z] attempt 2/5
[fetches 2019-11-05T17:10:19.446Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json to /builds/worker/fetches/jobs.json
[fetches 2019-11-05T17:10:19.446Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json
[fetches 2019-11-05T17:10:20.088Z] Download failed: HTTP Error 404: Not Found
[fetches 2019-11-05T17:10:20.088Z] sleeping for 89.00s (attempt 2/5)
[fetches 2019-11-05T17:11:49.134Z] attempt 3/5
[fetches 2019-11-05T17:11:49.134Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json to /builds/worker/fetches/jobs.json
[fetches 2019-11-05T17:11:49.135Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json
[fetches 2019-11-05T17:11:49.691Z] Download failed: HTTP Error 404: Not Found
[fetches 2019-11-05T17:11:49.691Z] sleeping for 136.00s (attempt 3/5)
[fetches 2019-11-05T17:14:05.703Z] attempt 4/5
[fetches 2019-11-05T17:14:05.703Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json to /builds/worker/fetches/jobs.json
[fetches 2019-11-05T17:14:05.704Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json
[fetches 2019-11-05T17:14:06.263Z] Download failed: HTTP Error 404: Not Found
[fetches 2019-11-05T17:14:06.264Z] sleeping for 203.50s (attempt 4/5)
[fetches 2019-11-05T17:17:29.777Z] attempt 5/5
[fetches 2019-11-05T17:17:29.777Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json to /builds/worker/fetches/jobs.json
[fetches 2019-11-05T17:17:29.777Z] Downloading https://queue.taskcluster.net/v1/task/NdGpByE6T--1rkKwcsiFHA/artifacts/public/test_info/jobs.json
[fetches 2019-11-05T17:17:30.416Z] Download failed: HTTP Error 404: Not Found
[fetches 2019-11-05T17:17:30.417Z] Traceback (most recent call last):
[fetches 2019-11-05T17:17:30.417Z]   File "/builds/worker/bin/fetch-content", line 666, in <module>
[fetches 2019-11-05T17:17:30.417Z]     sys.exit(main())
[fetches 2019-11-05T17:17:30.418Z]   File "/builds/worker/bin/fetch-content", line 662, in main
[fetches 2019-11-05T17:17:30.418Z]     return args.func(args)
[fetches 2019-11-05T17:17:30.418Z]   File "/builds/worker/bin/fetch-content", line 609, in command_task_artifacts
[fetches 2019-11-05T17:17:30.418Z]     fetch_urls(downloads)
[fetches 2019-11-05T17:17:30.418Z]   File "/builds/worker/bin/fetch-content", line 482, in fetch_urls
[fetches 2019-11-05T17:17:30.419Z]     f.result()
[fetches 2019-11-05T17:17:30.419Z]   File "/usr/lib/python3.5/concurrent/futures/_base.py", line 405, in result
[fetches 2019-11-05T17:17:30.419Z]     return self.__get_result()
[fetches 2019-11-05T17:17:30.419Z]   File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
[fetches 2019-11-05T17:17:30.419Z]     raise self._exception
[fetches 2019-11-05T17:17:30.420Z]   File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
[fetches 2019-11-05T17:17:30.420Z]     result = self.fn(*self.args, **self.kwargs)
[fetches 2019-11-05T17:17:30.420Z]   File "/builds/worker/bin/fetch-content", line 461, in fetch_and_extract
[fetches 2019-11-05T17:17:30.420Z]     download_to_path(url, dest_path, sha256=sha256, size=size)
[fetches 2019-11-05T17:17:30.421Z]   File "/builds/worker/bin/fetch-content", line 241, in download_to_path
[fetches 2019-11-05T17:17:30.421Z]     raise Exception("Download failed, no more retries!")
[fetches 2019-11-05T17:17:30.421Z] Exception: Download failed, no more retries!
[fetches 2019-11-05T17:17:30.439Z] removing /builds/worker/fetches
[fetches 2019-11-05T17:17:30.439Z] finished
[taskcluster 2019-11-05 17:17:30.724Z] === Task Finished ===
[taskcluster 2019-11-05 17:17:30.852Z] Artifact "public/visual-metrics.tar.xz" not found at "/builds/worker/artifacts/visual-metrics.tar.xz"
[taskcluster 2019-11-05 17:17:31.486Z] Unsuccessful task run with exit code: 1 completed in 513.227 seconds

Comment 1

[Greg Mierzwinski [:sparky]]

I haven't seen this issue in a while so I'm marking this as resolved.