1258343 - [mozlog] UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position %bar%: ordinal not in range(128)

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Seen this in various Marionette based tests (see bug 1255986) when creating a HTML report via mozlog:

00:33:01     INFO -      super(MarionetteTestResult, self).addFailure(test, err)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\moztest\adapters\unit.py", line 140, in addFailure
00:33:01     INFO -      extra=extra)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\logtypes.py", line 47, in inner
00:33:01     INFO -      return f(self, data)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\structuredlog.py", line 346, in test_end
00:33:01     INFO -      self._log_data("test_end", data)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\structuredlog.py", line 200, in _log_data
00:33:01     INFO -      self._handle_log(log_data)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\structuredlog.py", line 210, in _handle_log
00:33:01     INFO -      handler(data)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\handlers\base.py", line 93, in __call__
00:33:01     INFO -      formatted = self.formatter(data)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\handlers\base.py", line 62, in __call__
00:33:01     INFO -      return self.inner(item)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\reader.py", line 68, in __call__
00:33:01     INFO -      return handler(data)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\formatters\html\html.py", line 94, in test_end
00:33:01     INFO -      self.make_result_html(data)
00:33:01     INFO -    File "C:\slave\test\build\venv\lib\site-packages\mozlog\formatters\html\html.py", line 125, in make_result_html
00:33:01     INFO -      href = 'data:text/plain;charset=utf-8;base64,%s' % base64.b64encode(str(content).encode('utf-8'))
00:33:01    ERROR -  UnicodeEncodeError: 'ascii' codec can't encode character u'\xb0' in position 29763: ordinal not in range(128)

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/formatters/html/html.py#135

> href = 'data:text/plain;charset=utf-8;base64,%s' % base64.b64encode(str(content).encode('utf-8'))

Given that we expect UTF strings here, shouldn't we make the formatting string UTF too?

I feel this could be a good first bug. Joel, would you mentor this bug?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

this seems like a fixable thing, just reading this bug I would have no idea without researching mozlog and string encoding in python on what to do here.  Maybe this wouldn't be a good first bug, but possible a good next bug?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As said the fix would actually be:

> - href = 'data:text/plain;charset=utf-8;base64,%s' % base64.b64encode(str(content).encode('utf-8'))
> + href = u'data:text/plain;charset=utf-8;base64,%s' % base64.b64encode(str(content).encode('utf-8'))

This forces the formatting string to actually be a unicode one. Testing would still be necessary.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I would be happy to mentor this.

As an interested contributor, there is an example above of what needs to happen.  Your job will be to look through the rest of the code and ensure we are using unicdoe everywhere, verifying with unittests that this works, and writing some unittests to ensure we are covering this case.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: MozReview Request: Bug 1258343 - [mozlog] Fix HTML report generation for unicode characters. r=jmaher

Review commit: https://reviewboard.mozilla.org/r/51431/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/51431/

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8750470 [details]
MozReview Request: Bug 1258343 - [mozlog] Fix HTML report generation for unicode characters. r=jmaher

https://reviewboard.mozilla.org/r/51431/#review48371

thanks for the unittest with this!

Comment 7

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2d3c70bc4656

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/2d3c70bc4656

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Test-only change which we need to be landed on mozilla-aurora to get a regression fixed.

Comment 10

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/9ba3a7afa711

Comment 11

[Intermittent Failures Robot]

14 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 14

Platform breakdown:
* windows7-64: 8
* windows8-64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1258343&startday=2016-05-10&endday=2016-05-16&tree=all