1370057 - Intermittent socket.error: [Errno 104] Connection reset by peer after Checking if ... already exists (funsize-balrog-submitter.py)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=104278458&repo=mozilla-central
https://treeherder.mozilla.org/logviewer.html#?job_id=104282622&repo=mozilla-central

2017-06-03 13:52:31,750 - INFO - Starting new HTTPS connection (1): archive.mozilla.org
2017-06-03 13:52:31,773 - DEBUG - "GET /pub/firefox/nightly/2017/06/2017-06-03-03-02-04-mozilla-central-l10n/firefox-55.0a1.lv.mac.complete.mar HTTP/1.1" 200 65211458
2017-06-03 13:52:32,806 - DEBUG - Downloaded 65211458 bytes
2017-06-03 13:52:32,806 - DEBUG - Content-Length: 65211458 bytes
2017-06-03 13:52:32,808 - INFO - Verifying the signature...
2017-06-03 13:52:32,808 - INFO - Checking /tmp/tmpbKYT5V signature
2017-06-03 13:52:32,809 - DEBUG - index_offset is 65204030
2017-06-03 13:52:32,812 - DEBUG - first offset is 392
2017-06-03 13:52:32,812 - DEBUG - reading signatures
2017-06-03 13:52:32,812 - DEBUG - file_size: 65211458 bytes
2017-06-03 13:52:32,812 - DEBUG - 1 signatures present
2017-06-03 13:52:32,812 - DEBUG - signature data at 28 to 284
2017-06-03 13:52:32,812 - DEBUG - ver 1 signature 256 bytes at 20
2017-06-03 13:52:32,812 - DEBUG - 1 additional sections present
2017-06-03 13:52:32,813 - DEBUG - 1 104 bytes: 'firefox-mozilla-central\x0055.0a1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
2017-06-03 13:52:32,968 - INFO - Verification OK (RSA-PKCS1-SHA1)
2017-06-03 13:52:32,968 - INFO - Checking if mozilla-central/20170603030204/Firefox-mozilla-central-55.0a1-macosx64-lv.complete.mar already exists
Traceback (most recent call last):
  File "/home/worker/bin/funsize-balrog-submitter.py", line 207, in <module>
    main()
  File "/home/worker/bin/funsize-balrog-submitter.py", line 191, in main
    complete_mar_url, complete_mar_dest, args.signing_cert)
  File "/home/worker/bin/funsize-balrog-submitter.py", line 88, in verify_copy_to_s3
    if get_hash(key.get_contents_as_string()) == \
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1780, in get_contents_as_string
    response_headers=response_headers)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1648, in get_contents_to_file
    response_headers=response_headers)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1480, in get_file
    query_args=None)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1533, in _get_file_internal
    for bytes in self:
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 386, in next
    data = self.resp.read(self.BufferSize)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 413, in read
    return http_client.HTTPResponse.read(self, amt)
  File "/usr/lib/python2.7/httplib.py", line 611, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.7/socket.py", line 384, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
socket.error: [Errno 104] Connection reset by peer
[taskcluster 2017-06-03 13:52:33.776Z] === Task Finished ===
[taskcluster 2017-06-03 13:52:34.789Z] Unsuccessful task run with exit code: 1 completed in 162.212 seconds

Comment 1

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: adding retry

The default options seem sensible, and it's already imported, so this should be ok

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8874461 [details]
adding retry

lgtm!

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/32bf1cf6c7a8
add retry to safeguard network blips. r=mtabara a=release DONTBUILD

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8874461 [details]
adding retry

https://hg.mozilla.org/integration/mozilla-inbound/rev/32bf1cf6c7a8e96c662c5fac490212071fdef6e6

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #4)
> Comment on attachment 8874461 [details]
> adding retry
> 
> https://hg.mozilla.org/integration/mozilla-inbound/rev/
> 32bf1cf6c7a8e96c662c5fac490212071fdef6e6

Once this gets on central, we need to uplift to release branches.
Adding a :NI to keep this in mind.

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/32bf1cf6c7a8

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Wes Kocher (:KWierso) (PTO until June 1) from comment #6)
> https://hg.mozilla.org/mozilla-central/rev/32bf1cf6c7a8

Okay, the change is present in trunk now.
I was initially gonna start uplifting but then I realized we just merged beta -> release, which means this change will automatically get uplifted to beta on Monday. It can ride the trains to release after that I suppose.

Closing this for now, will re-open if it breaks anything.

Comment 8

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.