775264 - [bedrock][traceback] UnicodeDecodeError: 'utf8' codec can't decode byte 0xee in position 0: invalid continuation byte

Status: VERIFIED FIXED

(www.mozilla.org :: General, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

http://www.mozilla.org/ku/firefox/14.0.1/whatsnew/ triggers the following traceback:

Traceback (most recent call last):

  File "/data/www/www.mozilla.org-django/bedrock/vendor/lib/python/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/www.mozilla.org-django/bedrock/vendor/lib/python/django/views/decorators/vary.py", line 22, in inner_func
    response = func(*args, **kwargs)

  File "/data/www/www.mozilla.org-django/bedrock/apps/firefox/views.py", line 76, in whatsnew_redirect
    {'locales_with_video': locales_with_video})

  File "/data/www/www.mozilla.org-django/bedrock/lib/l10n_utils/__init__.py", line 38, in render
    return jingo.render(request, template, context, **kwargs)

  File "/data/www/www.mozilla.org-django/bedrock/vendor/src/jingo/jingo/__init__.py", line 80, in render
    rendered = render_to_string(request, template, context)

  File "/data/www/www.mozilla.org-django/bedrock/vendor/src/jingo/jingo/__init__.py", line 98, in render_to_string
    return template.render(**get_context())

  File "/usr/lib64/python2.6/site-packages/jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)

  File "/data/www/www.mozilla.org-django/bedrock/apps/firefox/templates/firefox/whatsnew.html", line 1, in top-level template code
    {% extends "/firefox/base-resp.html" %}

  File "/data/www/www.mozilla.org-django/bedrock/apps/firefox/templates/firefox/base-resp.html", line 1, in top-level template code
    {% extends "base-resp.html" %}

  File "/data/www/www.mozilla.org-django/bedrock/templates/base-resp.html", line 10, in top-level template code
    <title>{% block page_title_prefix %}Mozilla — {% endblock %}{% block page_title %}{% endblock %}{% block page_title_suffix %} — mozilla.org{% endblock %}</title>

  File "/data/www/www.mozilla.org-django/bedrock/apps/firefox/templates/firefox/whatsnew.html", line 6, in block "page_title"
    {% block page_title %}{{ _('Welcome to Firefox') }}{% endblock %}

  File "/data/www/www.mozilla.org-django/bedrock/lib/l10n_utils/helpers.py", line 34, in _
    trans = translate(text, ctx['request'].langfiles)

  File "/data/www/www.mozilla.org-django/bedrock/lib/l10n_utils/dotlang.py", line 62, in translate
    trans = parse(path)

  File "/data/www/www.mozilla.org-django/bedrock/lib/l10n_utils/dotlang.py", line 26, in parse
    for line in lines:

  File "/usr/lib64/python2.6/codecs.py", line 684, in next
    return self.reader.next()

  File "/usr/lib64/python2.6/codecs.py", line 615, in next
    line = self.readline()

  File "/usr/lib64/python2.6/codecs.py", line 530, in readline
    data = self.read(readsize, firstline=True)

  File "/usr/lib64/python2.6/codecs.py", line 477, in read
    newchars, decodedbytes = self.decode(data, self.errors)

UnicodeDecodeError: 'utf8' codec can't decode byte 0xee in position 0: invalid continuation byte

Comment 1

[Anthony Ricaud (:rik)]

Pascal: The error is in ku/main.lang. I don't know how but it has been messed up.

Comment 2

[Pascal Chevrel:pascalc]

It looks like the file as a buggy unicode line in the middle , I am going to fix but we need to file a follow up bug, we can't have the parser choke on a unicode character it doesn't understand and put the whole locale down, it can be intellectually satisfying but for our day to day operations, that's a major functional regression compared to our PHP parser...

Comment 3

[Pascal Chevrel:pascalc]

pushed in in r10758 on dev (http://www-dev.allizom.org/b/ku/firefox/whatsnew/),  pushed in r107660 on prod.

Comment 4

[Pascal Chevrel:pascalc]

That's a fix on the .lang file but the proper fix should be in the parser, morphing this bug into a bedrock l10n bug

Comment 5

[Pascal Chevrel:pascalc]

and the page is now back http://www.mozilla.org/ku/firefox/14.0.1/whatsnew/

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Pascal Chevrel:pascalc from comment #5)
> and the page is now back http://www.mozilla.org/ku/firefox/14.0.1/whatsnew/

Anything left to do here, then?  Can you take the bug and mark it fixed?

Comment 7

[Pascal Chevrel:pascalc]

We have fixed the file encoding but the parser should have been able to process it or ignore it without breaking the whole app, I am lowering the priority but keep this bug as an l10n webdev bug to fix in the platform.

Comment 8

[Anthony Ricaud (:rik)]

So I worked on this a bit.

Pascal: Would you prefer the fallback to be something like:
F��licitations ! Votre Firefox a ��t�� mis �� jour.

or fallback to the English sentence?

Comment 9

[Pascal Chevrel:pascalc]

I prefer the fallback with the black interrogation mark

Comment 10

[Anthony Ricaud (:rik)]

PR at https://github.com/mozilla/bedrock/pull/310

Comment 11

[[github robot]]

Commit pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/2988e08eafe091ae729177a3c94f581f3f4966b2
Don't fail on decoding errors.

fix bug 775264

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED on prod; thanks!