Intermittent partials concurrent.futures._base.TimeoutError

NEW
Assigned to

Status

Product:
Release Engineering
Component:
General
Type:
defect
Opened:
Last year
Updated:
Last year

People

(Reporter: nataliaCs, Assigned: sfraser)

Tracking

({leave-open})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Bug 1452927 Improve logging and retries for partials r=mtabara
46 bytes, text/x-phabricator-request
mtabara
: review+
Details | Review 
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=172849870&repo=mozilla-central

2018-04-10 12:05:33,157 - INFO - Downloading https://hg.mozilla.org/mozilla-central/raw-file/default/tools/update-packaging/unwrap_full_update.pl to /tmp/tmp1eqix4v3/unwrap_full_update.pl
2018-04-10 12:06:46,980 - INFO - Downloading https://hg.mozilla.org/mozilla-central/raw-file/default/tools/update-packaging/unwrap_full_update.pl to /tmp/tmp1eqix4v3/unwrap_full_update.pl
2018-04-10 12:08:12,510 - INFO - Downloading https://hg.mozilla.org/mozilla-central/raw-file/default/tools/update-packaging/unwrap_full_update.pl to /tmp/tmp1eqix4v3/unwrap_full_update.pl
2018-04-10 12:10:09,551 - INFO - Downloading https://hg.mozilla.org/mozilla-central/raw-file/default/tools/update-packaging/unwrap_full_update.pl to /tmp/tmp1eqix4v3/unwrap_full_update.pl
2018-04-10 12:12:49,807 - INFO - Downloading https://hg.mozilla.org/mozilla-central/raw-file/default/tools/update-packaging/unwrap_full_update.pl to /tmp/tmp1eqix4v3/unwrap_full_update.pl
2018-04-10 12:13:50,865 - WARNING - retry_async: <function download at 0x7f21cc158bf8>: too many retries!
Traceback (most recent call last):
  File "/home/worker/bin/funsize.py", line 504, in <module>
    main()
  File "/home/worker/bin/funsize.py", line 478, in main
    manifest = loop.run_until_complete(async_main(args, signing_certs))
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/home/worker/bin/funsize.py", line 405, in async_main
    await workenv.setup()
  File "/home/worker/bin/funsize.py", line 245, in setup
    await retry_download(url, dest=self.paths[filename], mode=0o755)
  File "/home/worker/bin/funsize.py", line 104, in retry_download
    kwargs=kwargs
  File "/usr/local/lib/python3.5/dist-packages/scriptworker/utils.py", line 252, in retry_async
    return await func(*args, **kwargs)
  File "/home/worker/bin/funsize.py", line 113, in download
    async with session.get(url, timeout=60) as resp:
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 690, in __aenter__
    self._resp = yield from self._coro
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 341, in _request
    break
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/helpers.py", line 727, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError
[taskcluster 2018-04-10 12:13:51.172Z] === Task Finished ===
[taskcluster 2018-04-10 12:13:51.236Z] Artifact "public/build/gu-IN/target.partial-2.mar" not found at "/home/worker/artifacts/target.partial-2.mar"
[taskcluster 2018-04-10 12:13:51.312Z] Artifact "public/build/gu-IN/target.partial-1.mar" not found at "/home/worker/artifacts/target.partial-1.mar"
[taskcluster 2018-04-10 12:13:51.368Z] Artifact "public/build/gu-IN/manifest.json" not found at "/home/worker/artifacts/manifest.json"
[taskcluster 2018-04-10 12:13:51.424Z] Artifact "public/build/gu-IN/target.partial-4.mar" not found at "/home/worker/artifacts/target.partial-4.mar"
[taskcluster 2018-04-10 12:13:51.488Z] Artifact "public/build/gu-IN/target.partial-3.mar" not found at "/home/worker/artifacts/target.partial-3.mar"
[taskcluster 2018-04-10 12:13:52.011Z] Unsuccessful task run with exit code: 1 completed in 527.45 seconds
See Also: → 1430600
Assignee: nobody → sfraser
These seem to be legitimate errors. The timing is spread out a bit, so I'm not convinced it's the number of concurrent tasks. I'll add more retry catching, but it feels like a band-aid over an underlying issue.
Attempt to get more information about download timeouts, and
also retry the partial generation if download timeouts happen too often.
Comment on attachment 8981797 [details]
Bug 1452927 Improve logging and retries for partials r=mtabara

Mihai Tabara [:mtabara]⌚️GMT has approved the revision.

https://phabricator.services.mozilla.com/D1467
Attachment #8981797 - Flags: review+
Pushed by sfraser@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d9afdede8e0f
Improve logging and retries for partials r=mtabara
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.