Closed Bug 1775165 Opened 3 years ago Closed 3 years ago

MSIX(push) frequently times out

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

https://treeherder.mozilla.org/jobs?repo=mozilla-release&searchStr=push%2Cmsix&revision=f1604ed27bc0e3853c5c120b321ba13226bc071c

(This has happened occasionally on beta too.)

I think I just need to increase the request timeout.

2022-06-20 21:03:28,503 - pushmsixscript.microsoft_store - INFO - >> creating a new submission...
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1371, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/app/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/app/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='manage.devcenter.microsoft.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/bin/pushmsixscript", line 8, in <module>
    sys.exit(main())
  File "/app/lib/python3.9/site-packages/pushmsixscript/script.py", line 44, in main
    return client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
  File "/app/lib/python3.9/site-packages/scriptworker_client/client.py", line 127, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, config, task))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/app/lib/python3.9/site-packages/scriptworker_client/client.py", line 182, in _handle_asyncio_loop
    await async_main(config, task)
  File "/app/lib/python3.9/site-packages/pushmsixscript/script.py", line 25, in async_main
    microsoft_store.push(config, msix_file_paths, channel, publish_mode)
  File "/app/lib/python3.9/site-packages/pushmsixscript/microsoft_store.py", line 64, in push
    _push_to_store(config, channel, msix_file_paths, publish_mode, access_token)
  File "/app/lib/python3.9/site-packages/pushmsixscript/microsoft_store.py", line 108, in _push_to_store
    (submission_request, encoding) = _create_submission(config, channel, session, headers)
  File "/app/lib/python3.9/site-packages/pushmsixscript/microsoft_store.py", line 143, in _create_submission
    response = session.post(url, headers=headers, timeout=int(config["request_timeout_seconds"]))
  File "/app/lib/python3.9/site-packages/requests/sessions.py", line 577, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/app/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/app/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/app/lib/python3.9/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='manage.devcenter.microsoft.com', port=443): Read timed out. (read timeout=30)
exit code: 1

https://github.com/mozilla-releng/scriptworker-scripts/pull/532 has landed in production. Hopefully new pushes will be more reliable.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Blocks: 1745203
You need to log in before you can comment on or make changes to this bug.