1364695 - Intermittent ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=98866763&repo=mozilla-inbound

07:57:24     INFO -   0:19.23 Downloading... 100.0 %
07:57:28     INFO -   0:23.48 Downloaded artifact to c:\builds\tooltool_cache\babc414ffc0457d27f5a1ed24a8e4873afbe2f1c1a4075469a27c005e1babc3b2a788f643f825efedff95b79686664c67ec4340ed535487168a3482e68559bc7
07:57:29     INFO -   0:24.83 hashed u'c:\\builds\\tooltool_cache\\babc414ffc0457d27f5a1ed24a8e4873afbe2f1c1a4075469a27c005e1babc3b2a788f643f825efedff95b79686664c67ec4340ed535487168a3482e68559bc7' with sha512 to be babc414ffc0457d27f5a1ed24a8e4873afbe2f1c1a4075469a27c005e1babc3b2a788f643f825efedff95b79686664c67ec4340ed535487168a3482e68559bc7
07:57:29     INFO -   0:24.83 Downloading clang.tar.bz2
07:57:29     INFO -   0:24.83 attempt 1/5
07:57:29     INFO -   0:24.83 Downloading to temporary location c:\builds\tooltool_cache\44dee70d525ea93952af27f943d1cc773311970c31d971d2bc2e3437cce0c899f3a03ddd8e42e86f1b4fd9ab1c4bc1767cdb0406eb4b3934ae4fc272dab830dc
07:57:30     INFO -  Error running mach:
07:57:30     INFO -      ['artifact', 'toolchain', '-v', '--retry', '4', '--tooltool-manifest', 'z:\\task_1494661831\\build\\src\\browser\\config\\tooltool-manifests\\win64\\clang.manifest', '--tooltool-url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', 'c:\\builds\\relengapi.tok', '--cache-dir', 'c:/builds/tooltool_cache']
07:57:30     INFO -  The error occurred in code that was called by the mach command. This is either
07:57:30     INFO -  a bug in the called code itself or in the way that mach is calling it.
07:57:30     INFO -  You should consider filing a bug for this issue.
07:57:30     INFO -  If filing a bug, please include the full output of mach, including this error
07:57:30     INFO -  message.
07:57:30     INFO -  The details of the failure are as follows:
07:57:30     INFO -  ConnectionError: ('Connection aborted.', BadStatusLine("''",))
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/mozbuild/mozbuild/mach_commands.py", line 1755, in artifact_toolchain
07:57:30     INFO -      record.fetch_with(cache)
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/mozbuild/mozbuild/mach_commands.py", line 1659, in fetch_with
07:57:30     INFO -      self.filename = cache.fetch(self.url)
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/mozbuild\mozbuild\artifacts.py", line 816, in fetch
07:57:30     INFO -      dl.wait()
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/dlmanager\dlmanager\manager.py", line 101, in wait
07:57:30     INFO -      self.raise_if_error()
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/dlmanager\dlmanager\manager.py", line 116, in raise_if_error
07:57:30     INFO -      six.reraise(*self.__error)
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/dlmanager\dlmanager\manager.py", line 157, in _download
07:57:30     INFO -      with closing(session.get(url, stream=True)) as response:
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/requests\requests\sessions.py", line 480, in get
07:57:30     INFO -      return self.request('GET', url, **kwargs)
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/requests\requests\sessions.py", line 468, in request
07:57:30     INFO -      resp = self.send(prep, **send_kwargs)
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/requests\requests\sessions.py", line 597, in send
07:57:30     INFO -      history = [resp for resp in gen] if allow_redirects else []
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/requests\requests\sessions.py", line 195, in resolve_redirects
07:57:30     INFO -      **adapter_kwargs
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/requests\requests\sessions.py", line 576, in send
07:57:30     INFO -      r = adapter.send(request, **kwargs)
07:57:30     INFO -    File "z:\task_1494661831\build\src\python/requests\requests\adapters.py", line 426, in send
07:57:30     INFO -      raise ConnectionError(err, request=request)
07:57:30    ERROR - Return code: 1
07:57:30    ERROR - 1 not in success codes: [0]
07:57:30  WARNING - setting return code to 2

Comment 1

[Intermittent Failures Robot]

5 failures in 879 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* windows2012-64: 4
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-05-08&endday=2017-05-14&tree=all

Comment 2

[Intermittent Failures Robot]

34 failures in 777 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #36 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 21
* mozilla-inbound: 7
* try: 4
* mozilla-central: 1
* graphics: 1

Platform breakdown:
* windows2012-32: 9
* windows2012-64: 7
* linux64-stylo: 5
* linux64: 4
* windows8-64: 3
* linux32: 2
* windowsxp: 1
* osx-10-7: 1
* android-4-2-x86: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-05-15&endday=2017-05-21&tree=all

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This hits again, this time massively: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=367734cc9370f2528dc564921e3d678cb352f514&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
Closed trees for this.

Comment 4

[Rok Garbas [:garbas]]

one of the webheads of relengapi didn't have port 5432 open to listen to new database. this was fixed in Bug 1344364.

Comment 5

[Rok Garbas [:garbas]]

one sucessfull build (finally).

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=07facc83000c26e54cd48adabcc6530c96686497&filter-searchStr=Linux+debug+Executed+by+TaskCluster+build-linux%2Fdebug+tc(B)

Also reponse time dropped from ~700ms to ~400ms

Comment 6

[Intermittent Failures Robot]

177 failures in 147 pushes (1.204 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 117
* autoland: 41
* mozilla-central: 17
* try: 2

Platform breakdown:
* linux64: 62
* osx-10-7: 24
* linux32: 22
* windows2012-64: 14
* android-4-0-armv7-api15: 14
* linux64-stylo: 13
* windows2012-32: 10
* android-api-15-gradle: 5
* android-4-2-x86: 5
* windowsxp: 4
* windows8-64: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-05-22&endday=2017-05-22&tree=all

Comment 7

[Intermittent Failures Robot]

24 failures in 161 pushes (0.149 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 12
* autoland: 7
* try: 2
* mozilla-central: 2
* graphics: 1

Platform breakdown:
* windows2012-32: 8
* osx-10-7: 5
* linux64: 4
* windows2012-64: 3
* windows2012-64-qr: 1
* android-api-15-gradle: 1
* android-4-2-x86: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-05-24&endday=2017-05-24&tree=all

Comment 8

[Intermittent Failures Robot]

15 failures in 155 pushes (0.097 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 4
* try: 1

Platform breakdown:
* windows2012-64: 6
* windows2012-32: 3
* linux64: 3
* windows8-64: 1
* osx-10-7: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-05-26&endday=2017-05-26&tree=all

Comment 9

[Intermittent Failures Robot]

247 failures in 891 pushes (0.277 failures/push) were associated with this bug in the last 7 days. 

This is the #2 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 142
* autoland: 76
* mozilla-central: 21
* try: 5
* graphics: 3

Platform breakdown:
* linux64: 74
* windows2012-32: 33
* osx-10-7: 32
* windows2012-64: 27
* linux32: 23
* android-4-0-armv7-api15: 16
* linux64-stylo: 13
* windowsxp: 7
* android-api-15-gradle: 7
* android-4-2-x86: 7
* windows8-64: 6
* windows2012-64-qr: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-05-22&endday=2017-05-28&tree=all

Comment 10

[Geoff Brown [:gbrown]]

:garbas -- This is much better compared to the May 22 spike, but failures continue at a rate of a dozen or so per day. Do you think you'll be able to eliminate this failure? Would it be helpful to retry when this happens? (It looks like there is retry logic involved, but it is not utilized in this case - most failures happen on "attempt 1/5".)

Comment 11

[Rok Garbas [:garbas]]

:gbrown: looks like there is something going wrong with retry logic. i will give it a look.

Comment 12

[Intermittent Failures Robot]

15 failures in 184 pushes (0.082 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 4
* try: 1

Platform breakdown:
* windows2012-32: 4
* windows2012-64: 3
* osx-10-7: 3
* windowsxp: 2
* windows8-64: 1
* linux64: 1
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-05-31&endday=2017-05-31&tree=all

Comment 13

[Intermittent Failures Robot]

50 failures in 820 pushes (0.061 failures/push) were associated with this bug in the last 7 days. 

This is the #23 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 27
* mozilla-inbound: 13
* try: 5
* graphics: 4
* mozilla-central: 1

Platform breakdown:
* windows2012-32: 17
* windows2012-64: 8
* osx-10-7: 8
* windowsxp: 5
* windows8-64: 4
* linux64: 4
* windows2012-64-qr: 1
* android-5-0-aarch64: 1
* android-4-2-x86: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-05-29&endday=2017-06-04&tree=all

Comment 14

[Intermittent Failures Robot]

61 failures in 864 pushes (0.071 failures/push) were associated with this bug in the last 7 days. 

This is the #20 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 37
* mozilla-inbound: 17
* mozilla-central: 6
* try: 1

Platform breakdown:
* windows2012-32: 14
* windows2012-64: 12
* linux64: 10
* osx-10-7: 5
* linux64-stylo: 3
* android-api-15-gradle: 3
* android-4-0-armv7-api15: 3
* windowsxp: 2
* windows8-64: 2
* windows2012-64-noopt: 2
* linux32: 2
* android-5-0-aarch64: 2
* linux64-noopt: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-06-05&endday=2017-06-11&tree=all

Comment 15

[Rok Garbas [:garbas]]

Attachment: tooltool_retry_on_connection_error.patch

Retry currently only happens when ``requests.exceptions.HttpError`` exception is raised. I think retry should also happen on ``requests.exceptions.ConnectionError``.

Should I also add a timeout between each retries, (eg. one second)?

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8876734 [details] [diff] [review]
tooltool_retry_on_connection_error.patch

Review of attachment 8876734 [details] [diff] [review]:
-----------------------------------------------------------------

This is a valid solution so it earns r+.

But I think a better solution is to use the built-in retry logic in requests. See https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-requests-request for code patterns. Note how it is even possible to configure backoff intervals for the retry logic. Also, remember to .mount('https://') as well as 'http://'.

Comment 17

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #16)
> Comment on attachment 8876734 [details] [diff] [review]
> tooltool_retry_on_connection_error.patch
> 
> Review of attachment 8876734 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> This is a valid solution so it earns r+.
> 
> But I think a better solution is to use the built-in retry logic in
> requests. See
> https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-
> requests-request for code patterns. Note how it is even possible to
> configure backoff intervals for the retry logic. Also, remember to
> .mount('https://') as well as 'http://'.

The problem with using the built-in retry logic is that it won't retry for HTTP errors, and then you end up with two retry strategies.

Comment 18

[Mike Hommey [:glandium]]

Comment on attachment 8876734 [details] [diff] [review]
tooltool_retry_on_connection_error.patch

Review of attachment 8876734 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/mach_commands.py
@@ +1790,5 @@
>                                                       sleeptime=60)):
>                  try:
>                      record.fetch_with(cache)
> +                except (requests.exceptions.HTTPError,
> +                        requests.exceptions.ConnectionError) as e:

Note it might be worth being broader than ConnectionError and HTTPError here, and use RequestException. Although that might be too broad... maybe add Timeout only?

Comment 19

[Intermittent Failures Robot]

62 failures in 814 pushes (0.076 failures/push) were associated with this bug in the last 7 days. 

This is the #33 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 29
* mozilla-inbound: 19
* mozilla-central: 5
* mozilla-beta: 5
* try: 4

Platform breakdown:
* windows2012-64: 19
* windows2012-32: 13
* osx-10-7: 6
* linux64: 5
* windows2012-32-noopt: 4
* windows8-64: 3
* windowsxp: 2
* windows8-32: 1
* windows2012-64-noopt: 1
* toolchains: 1
* osx-10-7-noopt: 1
* linux64-stylo: 1
* linux32: 1
* android-api-15-gradle: 1
* android-5-0-aarch64: 1
* android-4-2-x86: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 20

[Intermittent Failures Robot]

16 failures in 151 pushes (0.106 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 9
* mozilla-inbound: 6
* mozilla-beta: 1

Platform breakdown:
* windows2012-64: 5
* windows2012-32: 5
* windows8-32: 1
* osx-10-7: 1
* linux64-stylo: 1
* linux64: 1
* linux32: 1
* android-5-0-aarch64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-06-19&endday=2017-06-19&tree=all

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/45b27cacb06e
Make `mach artifact toolchain` also retry on ConnectionError. r=gps

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7bc0766b6a76
Syntax fixup for bug 1364695 for bustage. r=me

Comment 23

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/45b27cacb06e
https://hg.mozilla.org/mozilla-central/rev/7bc0766b6a76

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/89bdf23f16ea

Comment 25

[Intermittent Failures Robot]

34 failures in 892 pushes (0.038 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 20
* mozilla-inbound: 9
* try: 3
* mozilla-beta: 2

Platform breakdown:
* windows2012-32: 10
* windows2012-64: 7
* osx-10-7: 4
* linux64: 3
* windows8-64: 2
* android-5-0-aarch64: 2
* windowsxp: 1
* windows8-32: 1
* toolchains: 1
* osx-10-7-noopt: 1
* linux64-stylo: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 26

[Intermittent Failures Robot]

1 failures in 720 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 1

Platform breakdown:
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 27

[Intermittent Failures Robot]

2 failures in 822 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 2

Platform breakdown:
* windowsxp: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 28

[Intermittent Failures Robot]

1 failures in 908 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-08-21&endday=2017-08-27&tree=all

Comment 29

[Intermittent Failures Robot]

2 failures in 939 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 2

Platform breakdown:
* windows2012-64-noopt: 1
* windows2012-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1364695&startday=2017-08-28&endday=2017-09-03&tree=all