800462 - CEF library and unicode

Status: RESOLVED FIXED

(Marketplace Graveyard :: Code Quality, defect)

Description

[Andy McKay]

Fails on trying to check the length of a value as shown in bug 791292. I believe the problem is here:

https://github.com/mozilla/cef/blob/master/cef.py#L256

For example:

>>> len(str(u'\u0627\u0644\u062a\u0637\u0628-news'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-4: ordinal not in range(128)

I'd be in favour of won't fixing this because we'll be using metlog. But at least now we've got a trail.

Comment 1

[Tarek Ziadé (:tarek)]

I suspect something that's passed to log_cef() is unicode, where the API expects string but does not enforce it. 

Maybe we should assert log_cef() only gets str-utf8 or convert any unicode passed in prior to building the message

Comment 2

[Tarek Ziadé (:tarek)]

I have fixed the issue at https://github.com/mozilla/cef/commit/d26bf29d603089b16847baf065d13d6a49b6ad24

Andy please double-check everything works on your side now - if it does I will release a new cef lib version

Comment 3

[Andy McKay]

I don't think so. If you look at the traceback:

10:06:53 django.request:ERROR Internal Server Error: /reviewers/receipt/issue/птт-news :/Users/andy/.virtualenvs/zamboni/lib/python2.7/site-packages/django/core/handlers/base.py:215
Traceback (most recent call last):
  File "/Users/andy/.virtualenvs/zamboni/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/Users/andy/sandboxes/zamboni/apps/addons/decorators.py", line 33, in wrapper
    return f(request, addon, *args, **kw)
  File "/Users/andy/sandboxes/zamboni/apps/amo/decorators.py", line 128, in wrapper
    response = func(*args, **kw)
  File "/Users/andy/sandboxes/zamboni/apps/amo/decorators.py", line 51, in wrapper
    return f(request, *args, **kw)
  File "/Users/andy/sandboxes/zamboni/mkt/receipts/views.py", line 181, in issue
    receipt_cef.log(request, addon, 'sign', 'Receipt signing for %s' % flavour)
  File "/Users/andy/sandboxes/zamboni/lib/cef_loggers.py", line 83, in log
    severity, environ, **kwargs)
  File "/Users/andy/sandboxes/zamboni/vendor/lib/python/cef.py", line 305, in log_cef
    msg = _format_msg(fields, kw)
  File "/Users/andy/sandboxes/zamboni/vendor/lib/python/cef.py", line 263, in _format_msg
    msg = _CEF_FORMAT % fields
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 245: ordinal not in range(128)

The is on line 263. Before the _len actually occurs. This is actually a different bug.

Comment 4

[Andy McKay]

Data in the CEF lib comes from two sources: kwargs passed in and data read from the environ. In comment 3, data read from the environ that is unicode is the issue.

Comment 5

[Tarek Ziadé (:tarek)]

CEF 0.5 released with the fix - controlled by Andy

Thanks for the data!

Comment 6

[Andy McKay]

Thanks for the fix Tarek.