566101 - [k] IndexError: string index out of range

Status: VERIFIED FIXED

(support.mozilla.org :: Forum, task, P1)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Traceback on http://support-stage-new.mozilla.com/en-US/forums/test-forum/2?page=2:

Traceback (most recent call last):

  File "/data/virtualenvs/kitsune/src/django/django/core/handlers/base.py", line 101, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/support-stage-new.mozilla.com/kitsune/apps/forums/views.py", line 92, in posts
    'posts': posts_, 'form': form})

  File "/data/virtualenvs/kitsune/src/jingo/jingo/__init__.py", line 61, in render
    rendered = render_to_string(request, template, context)

  File "/data/virtualenvs/kitsune/src/jingo/jingo/__init__.py", line 80, in render_to_string
    ret = template.render(**context)

  File "/data/virtualenvs/kitsune/lib64/python2.6/site-packages/jinja2/environment.py", line 669, in render
    return self.environment.handle_exception(exc_info, True)

  File "/data/www/support-stage-new.mozilla.com/kitsune/apps/forums/templates/posts.html", line 5, in top-level template code
    {% set crumbs = [(url('forums.forums'), _('Forums')),

  File "/data/www/support-stage-new.mozilla.com/kitsune/templates/common/base.html", line 2, in top-level template code
    {% extends "layout/base.html" %}

  File "/data/www/support-stage-new.mozilla.com/kitsune/templates/layout/base.html", line 40, in top-level template code
    {% block content_area %}

  File "/data/www/support-stage-new.mozilla.com/kitsune/templates/common/base.html", line 8, in block "content_area"
    {% block content %}

  File "/data/www/support-stage-new.mozilla.com/kitsune/apps/forums/templates/posts.html", line 29, in block "content"
    {{ post.content_parsed }}

  File "/data/virtualenvs/kitsune/lib64/python2.6/site-packages/jinja2/environment.py", line 350, in getattr
    return getattr(obj, attribute)

  File "/data/www/support-stage-new.mozilla.com/kitsune/apps/forums/models.py", line 120, in content_parsed
    return jinja2.Markup(parser.parse(self.content, False))

  File "/data/www/support-stage-new.mozilla.com/kitsune/apps/sumo/utils.py", line 56, in parse
    return self.wikimarkup.parse(text, showToc)

  File "/data/virtualenvs/kitsune/src/py-wikimarkup/wikimarkup/parser.py", line 2278, in parse
    return p.parse(text)

  File "/data/virtualenvs/kitsune/src/py-wikimarkup/wikimarkup/parser.py", line 1676, in parse
    text = self.removeHtmlTags(text)

  File "/data/virtualenvs/kitsune/src/py-wikimarkup/wikimarkup/parser.py", line 595, in removeHtmlTags
    text = self.removeHtmlComments(text)

  File "/data/virtualenvs/kitsune/src/py-wikimarkup/wikimarkup/parser.py", line 706, in removeHtmlComments
    while text[spaceEnd] == u' ':

IndexError: string index out of range

Comment 1

[James Socol [:jsocol, :james]]

Obviously, this should be a 404. We should try to fix this in the paginator if possible. I'll grab it and give it a shot on Tuesday.

Comment 2

[Ricky Rosario [:rrosario, :r1cky]]

The issue is not that the page doesn't exist. It seems like something got posted that the markup parser isn't handling too well. We probably need to get the data (replies for page 2)?

Comment 3

[James Socol [:jsocol, :james]]

In bug 567382 comment 1 I've asked IT to dump the forums_post table for us before resyncing support-stage-new's database after yesterday's push.

Comment 4

[James Socol [:jsocol, :james]]

http://github.com/pcraciunoiu/py-wikimarkup/blob/master/wikimarkup/parser.py#L706

^^ the offending line in the wiki parser.

Comment 5

[Paul Craciunoiu [:paulc]]

Let me know if you want me to take this.

Comment 6

[James Socol [:jsocol, :james]]

It looks like an opening <!-- without a closing --> is at fault here.

Playing with it locally, it seems like we can skip the wikiparser's removeHtmlComments completely and rely on Bleach, since html5lib seems to drop comments completely, especially unclosed ones.

!BUT! this also means we need to make sure that min_length is still respected even after the markup is processed (gotta parse it on save, check the length, and save the raw, unparsed version).

Comment 7

[Paul Craciunoiu [:paulc]]

Removed the unnecessary call to removeHtmlComments from the parser. That doesn't actually fix this bug but it's related, so meh:
http://github.com/pcraciunoiu/py-wikimarkup/commit/ad7d943bd5afa95128dddc84b3d0fcd15b0d68be

The fix for this bug was actually in bleach - for that, see bug 568317.

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED.