1016007 - mozregression aborts in the case of a network error

Status: RESOLVED WORKSFORME

(Testing :: mozregression, defect)

Description

[(no longer active)]

It would be much nicer if it retried:

===== Downloaded 100% =====
Installing nightly
Starting nightly (revision: c7c7a42c67d3)
Was this inbound build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry', or 'exit' and press Enter): bad
Testing inbound build with timestamp 1400722939, revision dfd0c53bdf28cd37f0d9a3a2a5dbc02eac93bfd9
Downloading build from: http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1400722939/firefox-32.0a1.en-US.mac.dmg
Traceback (most recent call last):
  File "/usr/local/bin/mozregression", line 8, in <module>
    load_entry_point('mozregression==0.18', 'console_scripts', 'mozregression')()
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 276, in cli
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 182, in bisect_nightlies
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 182, in bisect_nightlies
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 182, in bisect_nightlies
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 185, in bisect_nightlies
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 185, in bisect_nightlies
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 159, in bisect_nightlies
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 134, in bisect_inbound
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 136, in bisect_inbound
  File "build/bdist.macosx-10.9-intel/egg/mozregression/regression.py", line 112, in bisect_inbound
  File "build/bdist.macosx-10.9-intel/egg/mozregression/runnightly.py", line 281, in start
  File "build/bdist.macosx-10.9-intel/egg/mozregression/runnightly.py", line 274, in install
  File "build/bdist.macosx-10.9-intel/egg/mozregression/runnightly.py", line 104, in download
  File "build/bdist.macosx-10.9-intel/egg/mozregression/utils.py", line 52, in download_url
  File "/Library/Python/2.7/site-packages/requests-2.2.1-py2.7.egg/requests/models.py", line 616, in generate
    decode_content=True):
  File "/Library/Python/2.7/site-packages/requests-2.2.1-py2.7.egg/requests/packages/urllib3/response.py", line 236, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/Library/Python/2.7/site-packages/requests-2.2.1-py2.7.egg/requests/packages/urllib3/response.py", line 183, in read
    data = self._fp.read(amt)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
socket.error: [Errno 54] Connection reset by peer
Exception OSError: (2, 'No such file or directory', 'firefox-32.0a1.en-US.mac.dmg') in <bound method FirefoxInbound.cleanup of <mozregression.runinbound.FirefoxInbound object at 0x10f9fb810>> ignored
Exception OSError: (2, 'No such file or directory', 'firefox-32.0a1.en-US.mac.dmg') in <bound method FirefoxNightly.cleanup of <mozregression.runnightly.FirefoxNightly object at 0x11021edd0>> ignored

Comment 1

[William Lachance (:wlach)]

Yes, we could have some kind of console interface offering to either retry the failed operation, similar to the interface to choosing the state of the build.

Comment 2

[Bruce Miller]

A related issue: I get
Traceback (most recent call last):
  File "/usr/bin/mozregression", line 9, in <module>
    load_entry_point('mozregression==0.19', 'console_scripts', 'mozregression')()
  File "build/bdist.linux-x86_64/egg/mozregression/regression.py", line 325, in cli
  File "build/bdist.linux-x86_64/egg/mozregression/regression.py", line 208, in bisect_nightlies
  File "build/bdist.linux-x86_64/egg/mozregression/regression.py", line 185, in bisect_nightlies
  File "build/bdist.linux-x86_64/egg/mozregression/runnightly.py", line 320, in start
  File "build/bdist.linux-x86_64/egg/mozregression/runnightly.py", line 317, in install
  File "build/bdist.linux-x86_64/egg/mozregression/runnightly.py", line 133, in install
  File "/usr/lib/python2.7/site-packages/mozInstall-1.10-py2.7.egg/mozinstall/mozinstall.py", line 119, in install
    install_dir = mozfile.extract(src, dest)[0]
  File "/usr/lib/python2.7/site-packages/mozfile-1.1-py2.7.egg/mozfile/mozfile.py", line 98, in extract
    namelist = extract_tarball(src, dest)
  File "/usr/lib/python2.7/site-packages/mozfile-1.1-py2.7.egg/mozfile/mozfile.py", line 38, in extract_tarball
    namelist = bundle.getnames()
  File "/usr/lib64/python2.7/tarfile.py", line 1813, in getnames
    return [tarinfo.name for tarinfo in self.getmembers()]
  File "/usr/lib64/python2.7/tarfile.py", line 1805, in getmembers
    self._load()        # all members, we first have to
  File "/usr/lib64/python2.7/tarfile.py", line 2380, in _load
    tarinfo = self.next()
  File "/usr/lib64/python2.7/tarfile.py", line 2315, in next
    self.fileobj.seek(self.offset)
mozinstall.mozinstall.InstallError: Failed to install "/home/bruce/tmp/mozregression/firefox-29.0a1.en-US.linux-x86_64.tar.bz2"

which is possibly due to a timeout, or otherwise very slow server or network.  Not at all clear from the trace. Of course it would be very helpful if it could *resume* downloading when I restart mozregression.

And something that Python programmers often don't realize: As nice as Python's stack trace facilities are, they're no substitute for an actual, useful error message.

A minor aside: since you have github Issues turned off for mozregression, it would be helpful to put a link to the bugzilla in the README.md.... thanks.

Comment 3

[William Lachance (:wlach)]

(In reply to Bruce Miller from comment #2)
> A related issue: I get
> ...
> 
> which is possibly due to a timeout, or otherwise very slow server or
> network.  Not at all clear from the trace. Of course it would be very
> helpful if it could *resume* downloading when I restart mozregression.
> 
> And something that Python programmers often don't realize: As nice as
> Python's stack trace facilities are, they're no substitute for an actual,
> useful error message.

I at least program in more languages than just Python. :) I know the current behaviour isn't that great, that's why this bug is still open. 

> A minor aside: since you have github Issues turned off for mozregression, it
> would be helpful to put a link to the bugzilla in the README.md.... thanks.

Yeah, good point. I'll fix this.

Comment 4

[Bruce Miller]

(In reply to William Lachance (:wlach) from comment #3)
> (In reply to Bruce Miller from comment #2)
> I at least program in more languages than just Python. :) I know the current
> behaviour isn't that great, that's why this bug is still open. 

Let me apologize for the snarkiness; it's an arduous process when the network
isn't cooperating, but it *is* a pretty cool tool for isolating a bug!

Comment 5

[nithin[:lazybug]]

hi, I am interested in fixing this, however I would need a bit of help with getting around the codebase

Comment 6

[Julien Pagès (:parkouss)]

Hello nithin,

Sure, you can work on this if you're interested!

First if you don't know mozregression, have a look
here: http://mozilla.github.io/mozregression/
and here to contribute: http://mozilla.github.io/mozregression/contribute.html

There are links that will help you for a first contribution, and to understand what it can do.


So, this is a pretty old bug, but there are things that we can still do here I think, that is:

1. propose to retry the download/install in case of network error, or install error for a manual bisection
2. in case we got one error like these in automatic bisection, retry the download/install at max three times.
3. for automatic Bisection, if after 3 retry that does not work, raise an error like mozregression.error.DownloadError - this way the stack trace would be hidden for users.

All the relative code for this lives in : https://github.com/mozilla/mozregression/blob/master/mozregression/test_runner.py. This file is responsible of downloading, installing, executing the binaries then test them:

ManualTestRunner:  The manual mode (the default)
CommandTestRunner: The automatic mode (see http://mozilla.github.io/mozregression/documentation/automatic-bisection.html)

The error we should handle here are mozinstall.mozinstall.InstallError and requests.exceptions.RequestException. We have to handle these errors when launcher is created, ie:

https://github.com/mozilla/mozregression/blob/741bfe1f7bdce138cb1e4211ee1fe907ad32ab30/mozregression/test_runner.py#L130
https://github.com/mozilla/mozregression/blob/741bfe1f7bdce138cb1e4211ee1fe907ad32ab30/mozregression/test_runner.py#L164

I assign you the bug - please ask if you have further questions.

@Will
Please have a look at the proposal here, and tell us if that's good for you. :)

Comment 7

[William Lachance (:wlach)]

All of this makes sense to me!

Comment 8

[Julien Pagès (:parkouss)]

Hmm, this is quite an old bug, and things have changed since last updated here. For example we now fixed the possible install errors with bug 1192485, and we are retrying at most three times most of our http requests.

So I'm going to close that bug, please reopen a new one if there are issues related to that.