urllib.quote does not work with non-ASCII characters

VERIFIED FIXED in 1.0

Status

Websites
spark.mozilla.org
--
blocker
VERIFIED FIXED
7 years ago
7 years ago

People

(Reporter: stas, Assigned: Franck Lecollinet)

Tracking

unspecified

Details

(Reporter)

Description

7 years ago
In desktop.views.home Python's standard urllib.quote is used to URL-encode certain strings, like the default tweet:

    'twitter_msg': urllib.quote(unicode(TWITTER_SHARE_MSG)),

This breaks the site when TWITTER_SHARE_MSG contains non-ASCII characters.  This is the case for any characters that use any sort of diacritics, be it French, Italian, German, Polish etc.  The Spanish inverted exclamation point "¡" is also affected.  Japanese is affected as well, for obvious reasons.

From http://docs.djangoproject.com/en/dev/ref/unicode/#uri-and-iri-handling

"The functions django.utils.http.urlquote() and django.utils.http.urlquote_plus() are versions of Python’s standard urllib.quote() and urllib.quote_plus() that work with non-ASCII characters. (The data is converted to UTF-8 prior to encoding.)"

I'm sorry that I didn't notice this earlier, but DEBUG was off on stage and I thought that the 500s were caused by errors in translating the %(placables)s in the strings.

I'm afraid this is a blocker bug.  At present, only en-US is sure to work.
there are 23 use cases http://pastebin.mozilla.org/1193485 and it seems that around 17 of them should be using django's urlquote http://pastebin.mozilla.org/1193497
(Assignee)

Updated

7 years ago
Assignee: nobody → franck.bugzilla
Target Milestone: --- → 1.0
(Assignee)

Comment 2

7 years ago
I replaced all listed calls to urllib.quote by Django's urlquote, except for line 3 in http://pastebin.mozilla.org/1193497. It's in a middleware taken from kitsune/zamboni that is already encoding to UTF-8 prior to quoting.

Commit:
https://github.com/mozilla/spark/commit/ce0a90ffaf22cf9e32148e4a3e3ff212ff4fcfc6
(Assignee)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 3

7 years ago
Thanks Franck.  I can see the locales back on stage now: 

https://spark.allizom.org/de/home
https://spark.allizom.org/fr/home
https://spark.allizom.org/it/home
https://spark.allizom.org/es/home
https://spark.allizom.org/pl/home
https://spark.allizom.org/vi/home

The only exception is Japanese

https://spark.allizom.org/ja/home

but I'm pretty sure it's a PO error this time, which I already fixed, but maybe the file hasn't been refreshed.
Good on both prod and staging; verified FIXED.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.