808590 - mozautolog throws UnicodeDecodeErrors submitting logs

Status: RESOLVED FIXED

(Testing Graveyard :: Autolog, defect)

Description

[Bob Clary [:bc] (inactive)]

On OSX 10.7 with Python 2.7.3 I get UnicodeDecodeErrors when attempting to submit logs to autolog via mozauto.log

2012-11-04 20:11:34,155|ERROR|Traceback (most recent call last):
  File "/work/mozilla/virtualenvwrapper/autophone/autophone/tests/robocoptest.py", line 272, in runjob
    testgroup.submit()
  File "/work/mozilla/virtualenvwrapper/autophone/mozautolog/mozautolog/mozautolog.py", line 743, in submit
    response_stream = urllib2.urlopen(req)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1207, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1174, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 958, in request
    self._send_request(method, url, body, headers)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 992, in _send_request
    self.endheaders(body)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 954, in endheaders
    self._send_output(message_body)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 812, in _send_output
    msg += message_body
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte

The problem is in httplib's _send_output

    def _send_output(self, message_body=None):
        """Send the currently buffered request and clear the buffer.

        Appends an extra \\r\\n to the buffer.
        A message_body may be specified, to be appended to the request.
        """
        self._buffer.extend(("", ""))
        msg = "\r\n".join(self._buffer)
        del self._buffer[:]
        # If msg and message_body are sent in a single send() call,
        # it will avoid performance problems caused by the interaction
        # between delayed ack and the Nagle algorithm.
        if isinstance(message_body, str):
            msg += message_body
>> We raise the error here since message_body is a str and we
>> run into unicode decoding issues adding strings and a binary gzipped
>> buffer.
            message_body = None
        self.send(msg)
        if message_body is not None:
            #message_body was not a string (i.e. it is a file) and
            #we must run the risk of Nagle
            self.send(message_body)



We can fix this by converting the gzipped buffer to a bytearray before submitting.

diff --git a/mozautolog/mozautolog.py b/mozautolog/mozautolog.py
--- a/mozautolog/mozautolog.py
+++ b/mozautolog/mozautolog.py
@@ -733,17 +733,17 @@ class RESTfulAutologTestGroup(AutologTes
                                                   (self.restserver,
                                                    self.contentType,
                                                    obj.id,
                                                    self.read_index,
                                                    self.write_index,
                                                    obj.doc_type,
                                                    self.server)
         req = urllib2.Request(host,
-                              buffer.getvalue(),
+                              bytearray(buffer.getvalue()),
                               {'content-type': 'application/gzip'})
         response_stream = urllib2.urlopen(req)
         if self.contentType == 'application/json':
           response = json.loads(response_stream.read())
         else:
           response = yaml.load(response_stream.read())
         if 'url' in response:
           obj.logurl = response['url']

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: patch

Comment 2

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/rev/8a2b3de2fd1b

Comment 3

[Jonathan Griffin (:jgriffin)]

Deployed.