476635 - Try server not sending logs to tinderbox

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

We're now getting this error on sm-try-master:

2009-02-02 23:50:31-0800 [-] Unhandled Error 
        Traceback (most recent call last): 
          File "/tools/buildbot/lib/python2.5/site-packages/buildbot/process/base.py", line 597, in buildFinished 
            self.build_status.buildFinished() 
          File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/builder.py", line 1174, in buildFinished 
            w.callback(self) 
          File "/tools/Twisted-8.0.1//lib/python2.5/site-packages/twisted/internet/defer.py", line 242, in callback 
            self._startRunCallbacks(result) 
          File "/tools/Twisted-8.0.1//lib/python2.5/site-packages/twisted/internet/defer.py", line 307, in _startRunCallbacks 
            self._runCallbacks() 
        --- <exception caught here> --- 
          File "/tools/Twisted-8.0.1//lib/python2.5/site-packages/twisted/internet/defer.py", line 323, in _runCallbacks 
            self.result = callback(self.result, *args, **kw) 
          File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/builder.py", line 1696, in _buildFinished 
            w.buildFinished(name, s, results) 
          File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/mail.py", line 217, in buildFinished 
            return self.buildMessage(name, build, results) 
          File "/tools/buildbot/lib/python2.5/site-packages/buildbot/status/tinderbox.py", line 217, in buildMessage 
            tinderboxLogs += shortText 
        exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 132107: ordinal not in range(128) 

This looks to be caused from mixing unicode and regular strings.  The arguments that are passed in from sendchange look to be unicode strings (e.g. revision, identifier, who), and so when mixed with the regular string data of the logs, we get this error.

I'm not sure if the unicode is new since the upgrade, or if our logs are outputting something differently.

The net result of this is that builds never appear to finish from tinderbox, even though buildbot thinks they're fine.

Comment 1

[Chris AtLee [:catlee]]

So the cause of this is that the HgPoller pulling data from hg.m.o/try is returning change information as unicode.  Which is fine in itself, but occasionally gcc outputs unicode quote characters (e.g. \u2018) in warnings.

The log files are treated as unencoded strings, so when buildbot is constructing the email to send to tinderbox, it tries to append the unencoded string from the logs with some other text that includes unicode generated from the change information...and then explodes.

I'm not sure at this point what changed recently so that this combination of events is causing us problems.

I have a patch running on sm-staging-try-master that should fix this, just waiting for some builds to go through.

Comment 2

[Chris AtLee [:catlee]]

Attachment: Make sure we're using 'str' objects for revision and author by encoding unicode into UTF-8

This looks like it's working well on sm-staging-try-master.  We need to test this out elsewhere as well.

Comment 3

[Chris AtLee [:catlee]]

This is now running on the main try server, sm-try-master.

I need to test it out on staging-master as well before committing.

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 360414 [details] [diff] [review]
Make sure we're using 'str' objects for revision and author by encoding unicode into UTF-8

changeset:   191:f9b3862e5fef

Comment 5

[Axel Hecht [:Pike]]

Why isn't this fixed on the status side?

Comment 6

[Chris AtLee [:catlee]]

Fixing this on the status side would mean patching up buildbot's TinderboxMailNotifier at the very least, which isn't unicode aware.

Making buildbot unicode-safe is a much larger project.  It seemed better to give buildbot data in a format it could safely deal with.

Comment 7

[Axel Hecht [:Pike]]

Are bugs filed for that? At least upstream?

IMHO, this is a short sighted decision and is going to cause more trouble in the long term.