Intermittent BadStatusLine: ''

RESOLVED FIXED

Status

Release Engineering
Applications: MozharnessCore
RESOLVED FIXED
2 years ago
7 months ago

People

(Reporter: Treeherder Bug Filer, Assigned: gbrown)

Tracking

({intermittent-failure})

unspecified
intermittent-failure

Firefox Tracking Flags

(firefox52 fixed, firefox53 fixed, firefox54 fixed)

Details

(Whiteboard: [stockwell fixed])

Attachments

(1 attachment)

Comment 1

2 years ago
Bug 1300812 should have fixed this.

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED

Comment 3

2 years ago
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 3
* mozilla-central: 2

Platform breakdown:
* linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2016-10-17&endday=2016-10-23&tree=all

Comment 4

2 years ago
14 failures in 623 pushes (0.022 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 7
* mozilla-release: 6
* mozilla-beta: 1

Platform breakdown:
* osx-10-10: 14

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2016-11-21&endday=2016-11-27&tree=all

Comment 5

2 years ago
5 failures in 694 pushes (0.007 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 3
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 4
* android-api-15-gradle: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2016-11-28&endday=2016-12-04&tree=all

Comment 6

2 years ago
20 failures in 34 pushes (0.588 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-central: 12
* mozilla-inbound: 8

Platform breakdown:
* osx-10-10: 20

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2017-01-14&endday=2017-01-14&tree=all

Comment 7

2 years ago
32 failures in 722 pushes (0.044 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-central: 14
* mozilla-inbound: 9
* mozilla-aurora: 5
* autoland: 4

Platform breakdown:
* osx-10-10: 32

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2017-01-09&endday=2017-01-15&tree=all
this seems to have become a problem again, but it comes in spurts
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 9

2 years ago
27 failures in 690 pushes (0.039 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 18
* mozilla-central: 5
* mozilla-inbound: 3
* graphics: 1

Platform breakdown:
* osx-10-10: 25
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2017-01-16&endday=2017-01-22&tree=all
Assignee: nobody → gbrown
See Also: → bug 1303668
This stack seems typical:

04:11:03    FATAL - Uncaught exception: Traceback (most recent call last):
04:11:03    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 2028, in run
04:11:03    FATAL -     self.run_action(action)
04:11:03    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 1967, in run_action
04:11:03    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
04:11:03    FATAL -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 1907, in _possibly_run_method
04:11:03    FATAL -     return getattr(self, method_name)()
04:11:03    FATAL -   File "scripts/scripts/desktop_unittest.py", line 550, in download_and_extract
04:11:03    FATAL -     suite_categories=target_categories)
04:11:03    FATAL -   File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 580, in download_and_extract
04:11:03    FATAL -     self._download_and_extract_symbols()
04:11:03    FATAL -   File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 520, in _download_and_extract_symbols
04:11:03    FATAL -     self.symbols_url = self.query_symbols_url()
04:11:03    FATAL -   File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 207, in query_symbols_url
04:11:03    FATAL -     self._urlopen(symbols_url, timeout=120)
04:11:03    FATAL -   File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 302, in _urlopen
04:11:03    FATAL -     return urllib2.urlopen(url, **kwargs)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen
04:11:03    FATAL -     return _opener.open(url, data, timeout)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 406, in open
04:11:03    FATAL -     response = meth(req, response)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 519, in http_response
04:11:03    FATAL -     'http', request, response, code, msg, hdrs)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 438, in error
04:11:03    FATAL -     result = self._call_chain(*args)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
04:11:03    FATAL -     result = func(*args)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 625, in http_error_302
04:11:03    FATAL -     return self.parent.open(new, timeout=req.timeout)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 400, in open
04:11:03    FATAL -     response = self._open(req, data)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 418, in _open
04:11:03    FATAL -     '_open', req)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
04:11:03    FATAL -     result = func(*args)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 1215, in https_open
04:11:03    FATAL -     return self.do_open(httplib.HTTPSConnection, req)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/urllib2.py", line 1180, in do_open
04:11:03    FATAL -     r = h.getresponse(buffering=True)
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/httplib.py", line 1030, in getresponse
04:11:03    FATAL -     response.begin()
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/httplib.py", line 407, in begin
04:11:03    FATAL -     version, status, reason = self._read_status()
04:11:03    FATAL -   File "/tools/python27/lib/python2.7/httplib.py", line 371, in _read_status
04:11:03    FATAL -     raise BadStatusLine(line)
04:11:03    FATAL - BadStatusLine: ''
04:11:03    FATAL - Running post_fatal callback...
04:11:03    FATAL - Exiting -1

query_symbols_url() expects and handles various exceptions from httplib, but not this one.
Created attachment 8830949 [details] [diff] [review]
handle all exceptions

Everything should work out if we just handle this exception like the other exceptions. I'm not sure what else httplib can raise, so let's catch Exception.
Attachment #8830949 - Flags: review?(aki)

Comment 12

2 years ago
Comment on attachment 8830949 [details] [diff] [review]
handle all exceptions

This works, but is a bit heavy handed.

If we're mainly concerned about httplib errors, all httplib exceptions seem to be based off httplib.HTTPException, which means we'd catch all of them if we add that to the list.  
https://docs.python.org/2/library/httplib.html#httplib.HTTPException

Also, socket.timeout inherits socket.error which inherits IOError, and urllib2.HTTPError inherits urllib2.URLError, which inherits IOError.  So this line could be rewritten as

    except (IOError, httplib.HTTPException) as ex:

to catch all the current exceptions and add all httplib exceptions to the list.

(I'm getting this by 
> 855$ python2
> Python 2.7.12 (default, Nov 24 2016, 18:40:05)
> [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import urllib2
> >>> help(urllib2.URLError)
> 855$ python2
> Python 2.7.12 (default, Nov 24 2016, 18:40:05)
> [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import urllib2
> >>> help(urllib2.URLError)
)
Attachment #8830949 - Flags: review?(aki) → review+

Comment 13

2 years ago
(In reply to Aki Sasaki [:aki] from comment #12)

>     except (IOError, httplib.HTTPException) as ex:

We'd need to import httplib to do this without erroring.  Also, I'm fine with either approach (either your patch or my suggested approach).  Also also, sorry for the double paste; I got hit by bug 1334240 today.
Thanks aki. You make good points and I'm tempted to limit the exceptions as you suggest. I worry that non-httplib code might raise in _urlopen(), or the _urlopen() implementation may change over time, so I'm going to keep the heavy handed approach.

Comment 15

2 years ago
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3424385712cf
Guard against all exceptions mozharness query_symbols_url(); r=aki

Comment 17

2 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/0930f9ff734f
status-firefox52: --- → fixed

Comment 18

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/3424385712cf
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
status-firefox54: --- → fixed
Resolution: --- → FIXED

Comment 19

2 years ago
25 failures in 749 pushes (0.033 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 10
* autoland: 9
* mozilla-aurora: 3
* mozilla-beta: 2
* graphics: 1

Platform breakdown:
* osx-10-10: 17
* linux64: 6
* windows7-32: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2017-01-23&endday=2017-01-29&tree=all
Whiteboard: [stockwell fixed]
4 failures in 888 pushes (0.005 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2

Platform breakdown:
* osx-10-10: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 21

11 months ago
3 failures in 901 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 2
* macosx64-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2017-08-07&endday=2017-08-13&tree=all

Comment 22

11 months ago
2 failures in 949 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 2

Platform breakdown:
* osx-10-10: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2017-08-14&endday=2017-08-20&tree=all

Comment 23

7 months ago
1 failures in 590 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1301807&startday=2017-12-18&endday=2017-12-24&tree=all
You need to log in before you can comment on or make changes to this bug.