1762080 - High frequency aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer - failure summary is missing

Status: RESOLVED DUPLICATE of bug 1590599

(Release Engineering :: Release Automation, defect)

Description

[Iulian Moraru]

The failure summary is missing in treeherder.
Push where we first saw the failure
Failure log

2022-03-29 23:05:07,269 - signingscript.sign - DEBUG - sign_hash_with_autograph took 3.33s; RSS:130400 (+0)
2022-03-29 23:05:07,277 - signingscript.sign - DEBUG - sign_authenticode_zip took 6.46s; RSS:130400 (+0)
Traceback (most recent call last):
  File "/app/bin/signingscript", line 8, in <module>
    sys.exit(main())
  File "/app/lib/python3.9/site-packages/signingscript/script.py", line 82, in main
    return scriptworker.client.sync_main(async_main, default_config=get_default_config())
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 166, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, context))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.9/site-packages/signingscript/script.py", line 42, in async_main
    output_files = await sign(context, os.path.join(work_dir, path), path_dict["formats"], authenticode_comment=path_dict.get("comment"))
  File "/app/lib/python3.9/site-packages/signingscript/task.py", line 151, in sign
    output = await signing_func(context, output, fmt, **kwargs)
  File "/app/lib/python3.9/site-packages/signingscript/sign.py", line 95, in wrapped
    return await f(*args, **kwargs)
  File "/app/lib/python3.9/site-packages/signingscript/sign.py", line 1388, in sign_authenticode_zip
    [f.result() for f in done]
  File "/app/lib/python3.9/site-packages/signingscript/sign.py", line 1388, in <listcomp>
    [f.result() for f in done]
  File "/app/lib/python3.9/site-packages/signingscript/sign.py", line 95, in wrapped
    return await f(*args, **kwargs)
  File "/app/lib/python3.9/site-packages/signingscript/sign.py", line 1331, in sign_authenticode_file
    if not await winsign.sign.sign_file(
  File "/app/lib/python3.9/site-packages/winsign/sign.py", line 128, in sign_file
    sig = await winsign.timestamp.add_old_timestamp(sig, timestamp_url)
  File "/app/lib/python3.9/site-packages/winsign/timestamp.py", line 182, in add_old_timestamp
    ts = await get_old_timestamp(signature, timestamp_url)
  File "/app/lib/python3.9/site-packages/winsign/timestamp.py", line 136, in get_old_timestamp
    async with aiohttp.request(
  File "/app/lib/python3.9/site-packages/aiohttp/client.py", line 1186, in __aenter__
    self._resp = await self._coro
  File "/app/lib/python3.9/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/app/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/app/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
exit code: 1

Comment 1

[Iulian Moraru]

On autoland, so far, the only affected build is Bs(build-signing) and on m-c we have this failures.
The only affected platform by this issue is Windows 2012.
They usually pass on the 2nd/3rd retrigger.

Comment 2

[Aki Sasaki (not active)]

After debugging, this appears to be the timestamp server (timestamp.digicert.com) hiccuping.
I'm going to wrap the winsign.sign.sign_file call in retry_async for more robustness.

Comment 3

[Aki Sasaki (not active)]

I believe this is another incarnation of bug 1590599.