Closed Bug 1618498 Opened 6 years ago Closed 6 years ago

Perma [tier2] condprof Exception: Download failed, no more retries! after Download failed: HTTP Error 404: Not Found

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Version:
Version 3
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox75 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla75
Tracking Flags:
Tracking Status
firefox75 --- fixed

People

(Reporter: malexandru, Assigned: tarek)

References

Details

Attachments

(1 file)

Bug 1618498 - condprof bustage on macOS
47 bytes, text/x-phabricator-request
Details | Review

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=290534849&repo=mozilla-central&lineNumber=117

Raw log: https://firefoxci.taskcluster-artifacts.net/Krf71cjrSVST2CX53RqSXQ/0/public/logs/live_backing.log

[taskcluster:warn 2020-02-26T10:15:02.281Z] [mounts] Download /Users/cltbld/downloads/dIwYHUAWQ06zYnTTnntzpQ of url https://hg.mozilla.org/mozilla-central/raw-file/7f41334e10443f4f1c7426e86fb0cb7adfdf4d62/taskcluster/scripts/misc/fetch-content has SHA256 b94eee2fac8db892874834d3f292967a9779a304f95ad916a8c7ea347caaf4f6 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2020-02-26T10:15:02.281Z] [mounts] Creating directory /Users/cltbld/tasks/task_1582707746 with permissions 0700
[taskcluster 2020-02-26T10:15:02.281Z] [mounts] Copying /Users/cltbld/downloads/dIwYHUAWQ06zYnTTnntzpQ to /Users/cltbld/tasks/task_1582707746/fetch-content
[taskcluster 2020-02-26T10:15:02.281Z] Executing command 0: chmod '+x' run-task
[taskcluster 2020-02-26T10:15:02.284Z]    Exit Code: 0
[taskcluster 2020-02-26T10:15:02.284Z]    User Time: 597µs
[taskcluster 2020-02-26T10:15:02.284Z]  Kernel Time: 616µs
[taskcluster 2020-02-26T10:15:02.284Z]    Wall Time: 3.381316ms
[taskcluster 2020-02-26T10:15:02.284Z]       Result: SUCCEEDED
[taskcluster 2020-02-26T10:15:02.284Z] Executing command 1: /usr/local/bin/python3 run-task -- bash -cx '$CONDPROF_ROOT/virtualenv/virtualenv.py -p python3 $CONDPROF_ROOT && export PATH="$PATH:$MOZ_FETCHES_DIR/bin" && cd $CONDPROF_ROOT && bin/python3 condprof/runner.py --firefox $MOZ_FETCHES_DIR/target.dmg ../../archive'
[setup 2020-02-26T10:15:02.377Z] run-task started in /Users/cltbld/tasks/task_1582707746
[setup 2020-02-26T10:15:02.380Z] MOZ_FETCHES_DIR is /Users/cltbld/tasks/task_1582707746/fetches
[fetches 2020-02-26T10:15:02.380Z] fetching artifacts
[fetches 2020-02-26T10:15:02.380Z] executing ['/usr/local/opt/python/bin/python3.7', '-u', 'fetch-content', 'task-artifacts']
[fetches 2020-02-26T10:15:02.498Z] attempt 1/5attempt 1/5
[fetches 2020-02-26T10:15:02.498Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:15:02.499Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.common.tests.tar.gz
[fetches 2020-02-26T10:15:02.499Z] 
[fetches 2020-02-26T10:15:02.499Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:15:02.499Z] attempt 1/5
[fetches 2020-02-26T10:15:02.499Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.dmg to /Users/cltbld/tasks/task_1582707746/fetches/target.dmg
[fetches 2020-02-26T10:15:02.499Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz
[fetches 2020-02-26T10:15:02.499Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.dmg
[fetches 2020-02-26T10:15:02.500Z] fetch-content:180: DeprecationWarning: cafile, capath and cadefault are deprecated, use a custom context instead.
[fetches 2020-02-26T10:15:02.500Z]   with urllib.request.urlopen(url, cafile=certifi.where()) if certifi else urllib.request.urlopen(url) as fh:
[fetches 2020-02-26T10:15:02.617Z] Download failed: HTTP Error 404: Not FoundDownload failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:15:02.618Z] 
[fetches 2020-02-26T10:15:02.618Z] sleeping for 61.00s (attempt 1/5)
[fetches 2020-02-26T10:15:02.618Z] sleeping for 60.00s (attempt 1/5)
[fetches 2020-02-26T10:15:06.652Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.dmg resolved to 79891400 bytes with sha256 d59d5d745a26d5565518dde4bdadc190c0ed2f9bbb5e4831757ed78c2249e025 in 4.152s
[fetches 2020-02-26T10:16:02.638Z] attempt 2/5
[fetches 2020-02-26T10:16:02.638Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.common.tests.tar.gz
[fetches 2020-02-26T10:16:02.638Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz
[fetches 2020-02-26T10:16:02.754Z] Download failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:16:02.754Z] sleeping for 90.00s (attempt 2/5)
[fetches 2020-02-26T10:16:03.701Z] attempt 2/5
[fetches 2020-02-26T10:16:03.702Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:16:03.702Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:16:03.799Z] Download failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:16:03.799Z] sleeping for 90.00s (attempt 2/5)
[fetches 2020-02-26T10:17:32.790Z] attempt 3/5
[fetches 2020-02-26T10:17:32.791Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.common.tests.tar.gz
[fetches 2020-02-26T10:17:32.791Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz
[fetches 2020-02-26T10:17:32.887Z] Download failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:17:32.887Z] sleeping for 135.00s (attempt 3/5)
[fetches 2020-02-26T10:17:33.802Z] attempt 3/5
[fetches 2020-02-26T10:17:33.803Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:17:33.803Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:17:33.902Z] Download failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:17:33.902Z] sleeping for 136.00s (attempt 3/5)
[fetches 2020-02-26T10:19:48.023Z] attempt 4/5
[fetches 2020-02-26T10:19:48.024Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.common.tests.tar.gz
[fetches 2020-02-26T10:19:48.024Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz
[fetches 2020-02-26T10:19:48.125Z] Download failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:19:48.126Z] sleeping for 202.50s (attempt 4/5)
[fetches 2020-02-26T10:19:50.027Z] attempt 4/5
[fetches 2020-02-26T10:19:50.027Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:19:50.027Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:19:50.130Z] Download failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:19:50.130Z] sleeping for 201.50s (attempt 4/5)
[fetches 2020-02-26T10:23:10.650Z] attempt 5/5
[fetches 2020-02-26T10:23:10.652Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.common.tests.tar.gz
[fetches 2020-02-26T10:23:10.652Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.common.tests.tar.gz
[fetches 2020-02-26T10:23:10.751Z] Download failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:23:11.663Z] attempt 5/5
[fetches 2020-02-26T10:23:11.663Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz to /Users/cltbld/tasks/task_1582707746/fetches/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:23:11.663Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xo7g_O0mTPq6LA2AceS9mw/artifacts/public/build/target.condprof.tests.tar.gz
[fetches 2020-02-26T10:23:11.767Z] Download failed: HTTP Error 404: Not Found
[fetches 2020-02-26T10:23:11.767Z] Traceback (most recent call last):
[fetches 2020-02-26T10:23:11.768Z]   File "fetch-content", line 663, in <module>
[fetches 2020-02-26T10:23:11.768Z]     sys.exit(main())
[fetches 2020-02-26T10:23:11.768Z]   File "fetch-content", line 659, in main
[fetches 2020-02-26T10:23:11.768Z]     return args.func(args)
[fetches 2020-02-26T10:23:11.768Z]   File "fetch-content", line 606, in command_task_artifacts
[fetches 2020-02-26T10:23:11.768Z]     fetch_urls(downloads)
[fetches 2020-02-26T10:23:11.768Z]   File "fetch-content", line 482, in fetch_urls
[fetches 2020-02-26T10:23:11.768Z]     f.result()
[fetches 2020-02-26T10:23:11.769Z]   File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 435, in result
[fetches 2020-02-26T10:23:11.770Z]     return self.__get_result()
[fetches 2020-02-26T10:23:11.770Z]   File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
[fetches 2020-02-26T10:23:11.771Z]     raise self._exception
[fetches 2020-02-26T10:23:11.771Z]   File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
[fetches 2020-02-26T10:23:11.771Z]     result = self.fn(*self.args, **self.kwargs)
[fetches 2020-02-26T10:23:11.771Z]   File "fetch-content", line 461, in fetch_and_extract
[fetches 2020-02-26T10:23:11.771Z]     download_to_path(url, dest_path, sha256=sha256, size=size)
[fetches 2020-02-26T10:23:11.771Z]   File "fetch-content", line 241, in download_to_path
[fetches 2020-02-26T10:23:11.772Z]     raise Exception("Download failed, no more retries!")
[fetches 2020-02-26T10:23:11.772Z] Exception: Download failed, no more retries!
[fetches 2020-02-26T10:23:11.803Z] removing /Users/cltbld/tasks/task_1582707746/fetches
[fetches 2020-02-26T10:23:11.806Z] finished
[taskcluster 2020-02-26T10:23:11.815Z]    Exit Code: 1
[taskcluster 2020-02-26T10:23:11.815Z]    User Time: 690.101ms
[taskcluster 2020-02-26T10:23:11.815Z]  Kernel Time: 180.908ms
[taskcluster 2020-02-26T10:23:11.815Z]    Wall Time: 8m9.530324941s
[taskcluster 2020-02-26T10:23:11.815Z]       Result: FAILED
[taskcluster 2020-02-26T10:23:11.815Z] === Task Finished ===
[taskcluster 2020-02-26T10:23:11.815Z] Task Duration: 8m9.534006459s
[taskcluster:error] Uploading error artifact public/condprof from file archive with message "Could not read directory '/Users/cltbld/tasks/task_1582707746/archive'", reason "file-missing-on-worker" and expiry 2021-02-25T09:31:09.629Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/Users/cltbld/tasks/task_1582707746/archive'
[taskcluster:error] exit status 1
[taskcluster:error] file-missing-on-worker: Could not read directory '/Users/cltbld/tasks/task_1582707746/archive'

Tarek, could you please take a look?

Flags: needinfo?(tarek)
Assignee: nobody → tarek
Flags: needinfo?(tarek)

yes I saw that earlier today, it looks like the tests common file is not present in the OS build anymore, but it's present in the OS shippable build.
I have a fix and ran a try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=264168420890d3f4bc4d1dc06e95744662b6ec61

target.common.tests.tar.gz is only present in the shippable build

Pushed by tziade@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/49099d517f2d condprof bustage on macOS r=sparky

https://hg.mozilla.org/mozilla-central/rev/49099d517f2d

Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox75: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla75
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: