681351 - [traceback] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 0: ordinal not in range(128)

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

http://addons-dev.allizom.org/en-US/firefox/addon/three-wolf-moon-shirt/reviews/ is giving me:

Traceback (most recent call last):

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/addons-dev.allizom.org/zamboni/apps/addons/decorators.py", line 23, in wrapper
    return f(request, addon, *args, **kw)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django-mobility/mobility/decorators.py", line 23, in wrapper
    return f(request, *args, **kw)

  File "/data/www/addons-dev.allizom.org/zamboni/apps/reviews/views.py", line 83, in review_list
    return jingo.render(request, template, ctx)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/jingo/jingo/__init__.py", line 79, in render
    rendered = render_to_string(request, template, context)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/jingo/jingo/__init__.py", line 97, in render_to_string
    return template.render(**get_context())

  File "/usr/lib64/python2.6/site-packages/jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)

  File "/data/www/addons-dev.allizom.org/zamboni/apps/reviews/templates/reviews/review_list.html", line 1, in top-level template code
    {% extends "base.html" %}

  File "/data/www/addons-dev.allizom.org/zamboni/templates/base.html", line 110, in top-level template code
    {% block main_content %}

  File "/data/www/addons-dev.allizom.org/zamboni/templates/base.html", line 115, in block "main_content"
    {% block content %}{% endblock %}

  File "/data/www/addons-dev.allizom.org/zamboni/apps/reviews/templates/reviews/review_list.html", line 49, in block "content"
    {% block review_list %}

  File "/data/www/addons-dev.allizom.org/zamboni/apps/reviews/templates/reviews/review_list.html", line 55, in block "review_list"
    {% include "reviews/review.html" %}

  File "/data/www/addons-dev.allizom.org/zamboni/apps/reviews/templates/reviews/review.html", line 21, in top-level template code
    {% trans user=review.user|user_link, date=review.created|datetime %}

  File "/data/www/addons-dev.allizom.org/zamboni/apps/users/helpers.py", line 39, in user_link
    return jinja2.Markup(_user_link(user))

  File "/data/www/addons-dev.allizom.org/zamboni/apps/users/helpers.py", line 59, in _user_link
    user.get_url_path(), unicode(jinja2.escape(user.name)))

  File "/usr/lib64/python2.6/site-packages/jinja2/_markupsafe/_native.py", line 21, in escape
    return Markup(unicode(s)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 0: ordinal not in range(128)

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Also happening on http://addons-dev.allizom.org/en-US/firefox/addon/lightning-overload/reviews/

Comment 2

[Wil Clouser [:clouserw]]

That is odd.  Is this something with the new DC?

Comment 3

[Wil Clouser [:clouserw]]

I ask because I don't get a traceback at https://addons.allizom.org/en-US/firefox/addon/lightning-overload/reviews/

Comment 4

[Jeremy Orem [:oremj]]

You and Jeff have sudo on that dev server if you want to jump on to try to debug.

Comment 5

[Jeff Balogh (:jbalogh)]

 (<UserProfile: [Bad Unicode data]>, 5783939L)]

mysql> select username from users where id=5783939;
+----------+
| username |
+----------+
| ????     |
+----------+
1 row in set (0.00 sec)

mysql> set names utf8;
Query OK, 0 rows affected (0.01 sec)

mysql> select username from users where id=5783939;
+--------------+
| username     |
+--------------+
| 寒星斜坠 |
+--------------+
1 row in set (0.00 sec)

Comment 6

[Jeff Balogh (:jbalogh)]

Character settings in `SHOW VARIABLES` are the same. The new db has this in `SHOW CREATE TABLE users`:

ENGINE=InnoDB AUTO_INCREMENT=5862165 DEFAULT CHARSET=utf8 COLLATE=utf8_bin

vs the old one:

ENGINE=InnoDB AUTO_INCREMENT=5868777 DEFAULT CHARSET=utf8

Comment 7

[Jeff Balogh (:jbalogh)]

It looks like this may be happening in Python. Django is interpreting some strings as unicode and leaving others as byte strings:

In [51]: a.name
Out[51]: '\xe5\xaf\x92\xe6\x98\x9f\xe6\x96\x9c\xe5\x9d\xa0'

In [52]: b.name
Out[52]: u'\u041a\u0443\u0440\u043e\u0439'

Comment 8

[Jeff Balogh (:jbalogh)]

https://github.com/jbalogh/zamboni/commit/3374eef

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Thx, Jeff.  Verified FIXED.