554206 - [k] Paginator throws UnicodeEncodeError on non-ascii (?) characters

Status: VERIFIED FIXED

(support.mozilla.org :: General, defect, P1)

Description

[Paul Craciunoiu [:paulc]]

The paginator doesn't seem to like non-ascii characters, e.g. urls like:
/ru/search?w=3&q=%D0%A3%D1%81%D1%82%D0%B0%D0%BD%D0%BE%D0%B2%D0%BA%D0%B0
/fr/search?w=3&q=Gloss%C3%A1rio

Comment 1

[Paul Craciunoiu [:paulc]]

We should be using flatqs for building the URL anyway, in which case this problem can be fixed as part of flatqs.

I've found these resources that should help:
http://stackoverflow.com/questions/804336/best-way-to-convert-a-unicode-url-to-ascii-utf-8-percent-escaped-in-python
http://stackoverflow.com/questions/1077564/python-unicode-unicodeencodeerror

Comment 3

[James Socol [:jsocol, :james]]

I've been working on this but every query with an accent guaranteed to return more than a page of results runs into bug 554210.

Comment 4

[James Socol [:jsocol, :james]]

I did push a new version of flatqs that should handle UTF-8 a little better:

http://github.com/jsocol/flatqs/commit/fc0b5d89cc78b894eaf73c48dde81d0f197c7a15

Comment 5

[James Socol [:jsocol, :james]]

Pull requests sent for r? and f?

http://github.com/jsocol/kitsune/commit/5bd11d7fb9d07088d9351288cafafd76880e28f1

Comment 6

[James Socol [:jsocol, :james]]

Pull requests sent for r? and f? (again)

http://github.com/jsocol/kitsune/commit/862128cb64793c2db9b0641bf39dba32972b921e

Comment 7

[Paul Craciunoiu [:paulc]]

This looks good to go. r+ at latest commit:
http://github.com/jsocol/kitsune/commit/f2ddcafb6de08bef55760993be8cfa5630f5c8b2

Comment 8

[James Socol [:jsocol, :james]]

Wooot!

http://github.com/jsocol/kitsune/commit/240c9469ee4f5928b2ca1b85e7b15f36229ed778

Build #28.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

I'm getting 500 Internal Server Errors with this:

http://support-stage-new.mozilla.com/ru/search?w=3&q=%D0%A3%D1%81%D1%82%D0%B0%D0%BD%D0%BE%D0%B2%D0%BA%D0%B0

Separate bug, or this?

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED on http://support-stage-new.mozilla.com/ru/search?w=3&q=%D0%A3%D1%81%D1%82%D0%B0%D0%BD%D0%BE%D0%B2%D0%BA%D0%B0.