564820 - UnicodeDecodeError in Django's CommonMiddleware

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect, P2)

Comment 1

[Jeff Balogh (:jbalogh)]

 'QUERY_STRING': 'src=\xc2\xbcwhscheck\xc2\xbe',
 'REQUEST_URI': '/z/en-US/firefox/addon/60?src=\xc2\xbcwhscheck\xc2\xbe',

Traceback (most recent call last):

 File "/data/virtualenvs/zamboni/lib/python2.6/site-packages/django/core/handlers/base.py", line 80, in get_response
   response = middleware_method(request)

 File "/data/virtualenvs/zamboni/lib/python2.6/site-packages/django/middleware/common.py", line 78, in process_request
   newurl += '?' + request.META['QUERY_STRING']

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 5: ordinal not in range(128)

Let's fix this bug in zamboni now and then tell Django about how they're broken.

Comment 2

[Jeff Balogh (:jbalogh)]

clouserw says this is from whitehat scanning.  It's not affecting real users, so it can wait.

Comment 3

[Wil Clouser [:clouserw]]

it's still some rookie shiz, we should fix it

Comment 4

[Jeff Balogh (:jbalogh)]

I can't reproduce this at the given url, and from what I can tell QUERY_STRING is always a byte string. 

clouserw: Do you know the exact url that whitehat is hitting us with here?

Comment 5

[Wil Clouser [:clouserw]]

(In reply to comment #4)
> I can't reproduce this at the given url, and from what I can tell QUERY_STRING
> is always a byte string. 
> 
> clouserw: Do you know the exact url that whitehat is hitting us with here?

The URL in comment 1 doesn't work because \x is denoting special characters.  They have a few in there that we ignore, but an umlaut is breaking us here.  Try:

https://preview.addons.mozilla.org/z/en-US/firefox/addon/60/?src=wenzellikes%C3%BC

Comment 6

[Jeff Balogh (:jbalogh)]

How do I turn these \x codes into characters that I can use to reproduce the error?

Another one: 

/z/ja/firefox/addon/9591?collection_uuid=6102d914-b53f-3ae7-a742-e744a003c044\xc3\xa3\xe2\x82\xac\xe2\x82\xac\xc3\xa3\xe2\x80\x9a\xc2\xb3\xc3\xa3\xc6\x92\xc2\xac\xc3\xa3\xc2\x81\xc2\xa7\xc3\xaf\xc2\xbc\xc2\xb4\xc3\xaf\xc2\xbd\xe2\x80\x94\xc3\xaf\xc2\xbd\xe2\x80\xb0\xc3\xaf\xc2\xbd\xe2\x80\x9d\xc3\xaf\xc2\xbd\xe2\x80\x9d\xc3\xaf\xc2\xbd\xe2\x80\xa6\xc3\xaf\xc2\xbd\xe2\x80\x99\xc3\xa3\xc6\x92\xe2\x80\x98\xc3\xa3\xc6\x92\xc2\xaf\xc3\xa3\xc6\x92\xc2\xbc\xc3\xa3\xe2\x80\x9a\xc2\xa2\xc3\xa3\xc6\x92\xc6\x92\xc3\xa3\xc6\x92\xe2\x80\x94\xc3\xa9\xe2\x80\x93\xe2\x80\x9c\xc3\xa9\xc2\x81\xe2\x80\xa2\xc3\xa3\xc2\x81\xe2\x80\x9e\xc3\xa3\xc2\x81\xc2\xaa\xc3\xa3\xc2\x81\xe2\x80\x94\xc3\xaf\xc2\xbc\xc2\x81

Comment 7

[James Socol [:jsocol, :james]]

>>> print '\xc3\xa3\xe2\x82\xac\xe2\x82\xac\xc3\xa3\xe2\x80\x9a\xc2\xb3\xc3\xa3\xc6\x92\xc2\xac\xc3\xa3\xc2\x81\xc2\xa7\xc3\xaf\xc2\xbc\xc2\xb4\xc3\xaf\xc2\xbd\xe2\x80\x94\xc3\xaf\xc2\xbd\xe2\x80\xb0\xc3\xaf\xc2\xbd\xe2\x80\x9d\xc3\xaf\xc2\xbd\xe2\x80\x9d\xc3\xaf\xc2\xbd\xe2\x80\xa6\xc3\xaf\xc2\xbd\xe2\x80\x99\xc3\xa3\xc6\x92\xe2\x80\x98\xc3\xa3\xc6\x92\xc2\xaf\xc3\xa3\xc6\x92\xc2\xbc\xc3\xa3\xe2\x80\x9a\xc2\xa2\xc3\xa3\xc6\x92\xc6\x92\xc3\xa3\xc6\x92\xe2\x80\x94\xc3\xa9\xe2\x80\x93\xe2\x80\x9c\xc3\xa9\xc2\x81\xe2\x80\xa2\xc3\xa3\xc2\x81\xe2\x80\x9e\xc3\xa3\xc2\x81\xc2\xaa\xc3\xa3\xc2\x81\xe2\x80\x94\xc3\xaf\xc2\xbc\xc2\x81'.decode('utf-8')

Comment 8

[Jeff Balogh (:jbalogh)]

http://github.com/jbalogh/zamboni/commits/896cfa62e

[qa-] since I only know how to reproduce it in tests, not in the browser.