1218763 - Antivirus should retry on SSLError

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

Attachment: SSLError.diff

04:56:18     INFO - Downloading pub/firefox/candidates/42.0-candidates/build1/update/win32/fa/firefox-40.0.3-42.0.partial.mar to cache/update/win32/fa/firefox-40.0.3-42.0.partial.mar
04:56:18    FATAL - Uncaught exception: Traceback (most recent call last):
04:56:18    FATAL -   File "/builds/slave/rel-m-rel-av-00000000000000000/scripts/mozharness/base/script.py", line 1698, in run
04:56:18    FATAL -     self.run_action(action)
04:56:18    FATAL -   File "/builds/slave/rel-m-rel-av-00000000000000000/scripts/mozharness/base/script.py", line 1640, in run_action
04:56:18    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
04:56:18    FATAL -   File "/builds/slave/rel-m-rel-av-00000000000000000/scripts/mozharness/base/script.py", line 1581, in _possibly_run_method
04:56:18    FATAL -     return getattr(self, method_name)()
04:56:18    FATAL -   File "scripts/scripts/release/antivirus.py", line 175, in get_files
04:56:18    FATAL -     pool.map(worker, find_release_files())
04:56:18    FATAL -   File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 227, in map
04:56:18    FATAL -     return self.map_async(func, iterable, chunksize).get()
04:56:18    FATAL -   File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 528, in get
04:56:18    FATAL -     raise self._value
04:56:18    FATAL - SSLError: The read operation timed out
04:56:18    FATAL - Running post_fatal callback...
04:56:18    FATAL - Exiting -1

Comment 1

[Rail Aliiev [:rail]]

Attachment: IOError.diff

This is a more generic one. IOError includes SSLError:

SSLError.mro()
[ssl.SSLError,
 socket.error,
 IOError,
 EnvironmentError,
 StandardError,
 Exception,
 BaseException,
 object]

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

https://hg.mozilla.org/releases/mozilla-esr38/rev/0eff77a5b7ba  (default for Firefox)
https://hg.mozilla.org/releases/mozilla-esr38/rev/84f6343a00e8  (THUNDERBIRD_38_VERBRANCH)

Comment 3

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/df5f057d1277

Comment 4

[Rail Aliiev [:rail]]

More errors:

22:16:27     INFO - Downloading pub/firefox/candidates/38.4.0esr-candidates/build2/update/win32/as/firefox-38.4.0esr.complete.mar to cache/update/win32/as/firefox-38.4.0esr.complete.mar
22:16:27    FATAL - Uncaught exception: Traceback (most recent call last):
22:16:27    FATAL -   File "/builds/slave/rel-m-esr38-av-000000000000000/scripts/mozharness/base/script.py", line 1693, in run
22:16:27    FATAL -     self.run_action(action)
22:16:27    FATAL -   File "/builds/slave/rel-m-esr38-av-000000000000000/scripts/mozharness/base/script.py", line 1635, in run_action
22:16:27    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
22:16:27    FATAL -   File "/builds/slave/rel-m-esr38-av-000000000000000/scripts/mozharness/base/script.py", line 1576, in _possibly_run_method
22:16:27    FATAL -     return getattr(self, method_name)()
22:16:27    FATAL -   File "scripts/scripts/release/antivirus.py", line 176, in get_files
22:16:27    FATAL -     pool.map(worker, find_release_files())
22:16:27    FATAL -   File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 227, in map
22:16:27    FATAL -     return self.map_async(func, iterable, chunksize).get()
22:16:27    FATAL -   File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 528, in get
22:16:27    FATAL -     raise self._value
22:16:27    FATAL - IncompleteRead: IncompleteRead(0 bytes read, 69074905 more expected)
22:16:27    FATAL - Running post_fatal callback...
22:16:27    FATAL - Exiting -1

Comment 5

[Rail Aliiev [:rail]]

Attachment: HTTPException.diff

httplib.IncompleteRead.__mro__

(httplib.IncompleteRead,
 httplib.HTTPException,
 Exception,
 BaseException,
 object)

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/df5f057d1277

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8679974 [details] [diff] [review]
HTTPException.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/16108f891268

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/16108f891268

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8679487 [details] [diff] [review]
IOError.diff

I'd like to land this patch to the release branches (it's already in aurora). It doesn't affect the product and used only by Releng to run antivirus checks as a part of release automation.

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8679974 [details] [diff] [review]
HTTPException.diff

I'd like to land this patch to the release branches (it's already in aurora). It doesn't affect the product and used only by Releng to run antivirus checks as a part of release automation.

Comment 11

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8679487 [details] [diff] [review]
IOError.diff

Approved for uplift to beta, release, and esr38; we need this for the build process

Comment 12

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8679974 [details] [diff] [review]
HTTPException.diff

Approved to land on beta, release, esr38

Comment 13

[Rail Aliiev [:rail]]

Thanks!

Comment 14

[Rail Aliiev [:rail]]

Beta
remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/40ae04af867c
remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/f2f176ef9b45

Comment 15

[Rail Aliiev [:rail]]

Release
remote:   https://hg.mozilla.org/releases/mozilla-release/rev/1d4ea45451a7
remote:   https://hg.mozilla.org/releases/mozilla-release/rev/c6ee37cf63fb

Comment 16

[Rail Aliiev [:rail]]

and ESR38:
https://hg.mozilla.org/releases/mozilla-esr38/rev/12d999e5fe3b

Comment 17

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.