Intermittent partials concurrent.futures._base.TimeoutError

Assigned to


Last year
Last year


(Reporter: nataliaCs, Assigned: sfraser)



Firefox Tracking Flags

(Not tracked)



(1 attachment)

Failure log:

2018-04-10 12:05:33,157 - INFO - Downloading to /tmp/tmp1eqix4v3/
2018-04-10 12:06:46,980 - INFO - Downloading to /tmp/tmp1eqix4v3/
2018-04-10 12:08:12,510 - INFO - Downloading to /tmp/tmp1eqix4v3/
2018-04-10 12:10:09,551 - INFO - Downloading to /tmp/tmp1eqix4v3/
2018-04-10 12:12:49,807 - INFO - Downloading to /tmp/tmp1eqix4v3/
2018-04-10 12:13:50,865 - WARNING - retry_async: <function download at 0x7f21cc158bf8>: too many retries!
Traceback (most recent call last):
  File "/home/worker/bin/", line 504, in <module>
  File "/home/worker/bin/", line 478, in main
    manifest = loop.run_until_complete(async_main(args, signing_certs))
  File "/usr/lib/python3.5/asyncio/", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/", line 241, in _step
    result = coro.throw(exc)
  File "/home/worker/bin/", line 405, in async_main
    await workenv.setup()
  File "/home/worker/bin/", line 245, in setup
    await retry_download(url, dest=self.paths[filename], mode=0o755)
  File "/home/worker/bin/", line 104, in retry_download
  File "/usr/local/lib/python3.5/dist-packages/scriptworker/", line 252, in retry_async
    return await func(*args, **kwargs)
  File "/home/worker/bin/", line 113, in download
    async with session.get(url, timeout=60) as resp:
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/", line 690, in __aenter__
    self._resp = yield from self._coro
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/", line 341, in _request
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/", line 727, in __exit__
    raise asyncio.TimeoutError from None
[taskcluster 2018-04-10 12:13:51.172Z] === Task Finished ===
[taskcluster 2018-04-10 12:13:51.236Z] Artifact "public/build/gu-IN/target.partial-2.mar" not found at "/home/worker/artifacts/target.partial-2.mar"
[taskcluster 2018-04-10 12:13:51.312Z] Artifact "public/build/gu-IN/target.partial-1.mar" not found at "/home/worker/artifacts/target.partial-1.mar"
[taskcluster 2018-04-10 12:13:51.368Z] Artifact "public/build/gu-IN/manifest.json" not found at "/home/worker/artifacts/manifest.json"
[taskcluster 2018-04-10 12:13:51.424Z] Artifact "public/build/gu-IN/target.partial-4.mar" not found at "/home/worker/artifacts/target.partial-4.mar"
[taskcluster 2018-04-10 12:13:51.488Z] Artifact "public/build/gu-IN/target.partial-3.mar" not found at "/home/worker/artifacts/target.partial-3.mar"
[taskcluster 2018-04-10 12:13:52.011Z] Unsuccessful task run with exit code: 1 completed in 527.45 seconds
See Also: → 1430600
Assignee: nobody → sfraser
These seem to be legitimate errors. The timing is spread out a bit, so I'm not convinced it's the number of concurrent tasks. I'll add more retry catching, but it feels like a band-aid over an underlying issue.
Attempt to get more information about download timeouts, and
also retry the partial generation if download timeouts happen too often.
Comment on attachment 8981797 [details]
Bug 1452927 Improve logging and retries for partials r=mtabara

Mihai Tabara [:mtabara]⌚️GMT has approved the revision.
Attachment #8981797 - Flags: review+
Pushed by
Improve logging and retries for partials r=mtabara
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.