645667 - urllib.quote does not work with non-ASCII characters

Status: VERIFIED FIXED

(Websites :: spark.mozilla.org, defect)

Description

[Staś Małolepszy :stas]

In desktop.views.home Python's standard urllib.quote is used to URL-encode certain strings, like the default tweet:

    'twitter_msg': urllib.quote(unicode(TWITTER_SHARE_MSG)),

This breaks the site when TWITTER_SHARE_MSG contains non-ASCII characters.  This is the case for any characters that use any sort of diacritics, be it French, Italian, German, Polish etc.  The Spanish inverted exclamation point "¡" is also affected.  Japanese is affected as well, for obvious reasons.

From http://docs.djangoproject.com/en/dev/ref/unicode/#uri-and-iri-handling

"The functions django.utils.http.urlquote() and django.utils.http.urlquote_plus() are versions of Python’s standard urllib.quote() and urllib.quote_plus() that work with non-ASCII characters. (The data is converted to UTF-8 prior to encoding.)"

I'm sorry that I didn't notice this earlier, but DEBUG was off on stage and I thought that the 500s were caused by errors in translating the %(placables)s in the strings.

I'm afraid this is a blocker bug.  At present, only en-US is sure to work.

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

there are 23 use cases http://pastebin.mozilla.org/1193485 and it seems that around 17 of them should be using django's urlquote http://pastebin.mozilla.org/1193497

Comment 2

[Franck Lecollinet]

I replaced all listed calls to urllib.quote by Django's urlquote, except for line 3 in http://pastebin.mozilla.org/1193497. It's in a middleware taken from kitsune/zamboni that is already encoding to UTF-8 prior to quoting.

Commit:
https://github.com/mozilla/spark/commit/ce0a90ffaf22cf9e32148e4a3e3ff212ff4fcfc6

Comment 3

[Staś Małolepszy :stas]

Thanks Franck.  I can see the locales back on stage now: 

https://spark.allizom.org/de/home
https://spark.allizom.org/fr/home
https://spark.allizom.org/it/home
https://spark.allizom.org/es/home
https://spark.allizom.org/pl/home
https://spark.allizom.org/vi/home

The only exception is Japanese

https://spark.allizom.org/ja/home

but I'm pretty sure it's a PO error this time, which I already fixed, but maybe the file hasn't been refreshed.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

Good on both prod and staging; verified FIXED.