596116 - Unicode filenames are not supported for image uploads

Status: VERIFIED FIXED

(support.mozilla.org :: General, defect, P2)

Description

[Paul Craciunoiu [:paulc]]

It looks like we don't support Unicode in our filenames for e.g. image uploads to questions.

Traceback (most recent call last):

  File "/data/virtualenvs/kitsune/src/django/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/virtualenvs/kitsune/src/django/django/contrib/auth/decorators.py", line 25, in _wrapped_view
    return view_func(request, *args, **kwargs)

  File "/data/virtualenvs/kitsune/src/django/django/views/decorators/http.py", line 37, in inner
    return func(request, *args, **kwargs)

  File "/data/www/support.mozilla.com/kitsune/apps/upload/views.py", line 38, in up_image_async
    files = upload_images(request, obj)

  File "/data/www/support.mozilla.com/kitsune/apps/upload/utils.py", line 51, in upload_images
    image = create_image_attachment(up_file, obj, request.user)

  File "/data/www/support.mozilla.com/kitsune/apps/upload/utils.py", line 19, in create_image_attachment
    image.file.save(up_file.name, file_, save=False)

  File "/data/virtualenvs/kitsune/src/django/django/db/models/fields/files.py", line 92, in save
    self.name = self.storage.save(name, content)

  File "/data/virtualenvs/kitsune/src/django/django/core/files/storage.py", line 47, in save
    name = self.get_available_name(name)

  File "/data/virtualenvs/kitsune/src/django/django/core/files/storage.py", line 73, in get_available_name
    while self.exists(name):

  File "/data/virtualenvs/kitsune/src/django/django/core/files/storage.py", line 196, in exists
    return os.path.exists(self.path(name))

  File "/data/virtualenvs/BASELINE/lib/python2.6/genericpath.py", line 18, in exists
    st = os.stat(path)

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe2' in position 61: ordinal not in range(128)


<WSGIRequest
GET:<QueryDict: {}>,
POST:<QueryDict: {u'csrfmiddlewaretoken': [u'26f720e709b9463b46f7996ab9de2e6e']}>,
COOKIES:{'SSID_SUMO': 'AwB_JykAAAAAD1WOTIaPGwEPVY5MAgDEfo5MAAAAAAAAAADEfo5MAQA0AAAAywAAAAI',
 'SSLB_SUMO': '1',
 'SSRT': 'xH6OTAA',
 'SSSC_SUMO': '3.G5516440116506890118.2.52.203',
 'SUMOloggedin': 'true',
 'SUMOv1': 'bc1k212or7r3ie4os8iktad2j4',
 'WT_FPC': 'id=69.165.164.153-3162148480.30102370:lv=1284392660461:ss=1284392583714',
 'csrftoken': '26f720e709b9463b46f7996ab9de2e6e',
 'multidb_pin_writes': 'y',
 'sessionid': 'c6e2d99bbca654ca8635fcffaae72230',
 'wtspl': '423396'},
META:{'CONTENT_LENGTH': '505898',
 'CONTENT_TYPE': 'multipart/form-data; boundary=---------------------------17954120783585',
 'CSRF_COOKIE': '26f720e709b9463b46f7996ab9de2e6e',
 'DOCUMENT_ROOT': '/data/www/support.mozilla.com/webroot',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTPS': 'on',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_COOKIE': 'WT_FPC=id=69.165.164.153-3162148480.30102370:lv=1284392660461:ss=1284392583714; wtspl=423396; SSID_SUMO=AwB_JykAAAAAD1WOTIaPGwEPVY5MAgDEfo5MAAAAAAAAAADEfo5MAQA0AAAAywAAAAI; SSRT=xH6OTAA; csrftoken=26f720e709b9463b46f7996ab9de2e6e; SSLB_SUMO=1; SSSC_SUMO=3.G5516440116506890118.2.52.203; SUMOv1=bc1k212or7r3ie4os8iktad2j4; SUMOloggedin=true; sessionid=c6e2d99bbca654ca8635fcffaae72230; multidb_pin_writes=y',
 'HTTP_HOST': 'support.mozilla.com',
 'HTTP_KEEP_ALIVE': '115',
 'HTTP_REFERER': 'https://support.mozilla.com/en-US/questions/750139',
 'HTTP_SSLCLIENTCERTSTATUS': 'NoClientCert',
 'HTTP_SSLCLIENTCIPHER': 'SSL_RSA_WITH_RC4_128_SHA, version=TLSv1, bits=128',
 'HTTP_SSLSESSIONID': 'F3D3506453AD630E9BC83B4375457532EAE5F92B59667D52D3ACAB325BF872CF',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.9) Gecko/20100824 Firefox/3.6.9',
 'HTTP_X_CLUSTER_CLIENT_IP': '76.10.134.138',
 'HTTP_X_FORWARDED_FOR': '76.10.134.138',
 'PATH_INFO': u'/en-US/upload/image/questions.Question/750139',
 'PATH_TRANSLATED': 'redirect:/k/en-US/upload/image/questions.Question/750139/upload/image/questions.Question/750139',
 'QUERY_STRING': '',
 'REDIRECT_HTTPS': 'on',
 'REDIRECT_SCRIPT_URI': 'http://support.mozilla.com/en-US/upload/image/questions.Question/750139',
 'REDIRECT_SCRIPT_URL': '/en-US/upload/image/questions.Question/750139',
 'REDIRECT_STATUS': '200',
 'REDIRECT_URL': '/en-US/upload/image/questions.Question/750139',
 'REDIRECT_is-forwarded': '1',
 'REMOTE_ADDR': '10.2.81.101',
 'REMOTE_PORT': '13171',
 'REQUEST_METHOD': 'POST',
 'REQUEST_URI': '/en-US/upload/image/questions.Question/750139',
 'SCRIPT_FILENAME': '/data/www/support.mozilla.com/kitsune/wsgi/kitsune.wsgi',
 'SCRIPT_NAME': u'',
 'SCRIPT_URI': 'http://support.mozilla.com/en-US/upload/image/questions.Question/750139',
 'SCRIPT_URL': '/en-US/upload/image/questions.Question/750139',
 'SERVER_ADDR': '10.2.81.56',
 'SERVER_ADMIN': '[e-mail]',
 'SERVER_NAME': 'support.mozilla.com',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SIGNATURE': '',
 'SERVER_SOFTWARE': 'Apache',
 'is-forwarded': '1',
 'mod_wsgi.application_group': 'support.mozilla.com|/k',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '81',
 'mod_wsgi.process_group': 'kitsune-ssl',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 2),
 'wsgi.errors': <mod_wsgi.Log object at 0xa17ad90>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x9ae6f50>,
 'wsgi.input': <mod_wsgi.Input object at 0xa73e688>,
 'wsgi.loaded': datetime.datetime(2010, 9, 12, 5, 38, 20, 867566),
 'wsgi.multiprocess': True,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 1)}>

Comment 1

[James Socol [:jsocol, :james]]

We talked about this in IRC and for attached images we can just munge the file name to whatever we want, while for media gallery images in the KB we can tell people when there's a problem with the name (part of regular old form validation).

But, Paul, unless this is already patched, I think it needs to slip to 2.3.

Comment 2

[Paul Craciunoiu [:paulc]]

(In reply to comment #1)
> But, Paul, unless this is already patched, I think it needs to slip to 2.3.
No patch. Fine with me.

Comment 4

[James Socol [:jsocol, :james]]

As far as we've been able to tell, this is environmentally dependent. It's either related to file-system, or distro configuration of Apache.

Comment 5

[Paul Craciunoiu [:paulc]]

https://github.com/jsocol/kitsune/commit/6c20d6c15ebf84db3b0260f9ada96892a8691936

Comment 6

[Rebecca Billings [:rbillings]]

Please add QA steps or [qa-], thanks

Comment 7

[Paul Craciunoiu [:paulc]]

You should be able to upload filenames with unicode in them. For best testing, I'd recommend trying both "high-ascii" (e.g. French file name including non-ascii chars) and "high-unicode" (e.g. Japanese file name).

This bug also adds form validation messages and limits filename length explicitly. You should be unable to upload files with names longer than 200 characters.

Comment 8

[Paul Craciunoiu [:paulc]]

To clarify, this applies to both gallery uploads as well as image uploads to questions.

Comment 9

[Paul Craciunoiu [:paulc]]

Looks like this still doesn't work on master.support. Seeing stack traces.

Comment 10

[Rebecca Billings [:rbillings]]

I am unable to upload images with high-ascii or high-unicode filenames in the media gallery- the submit file button is disabled and it  never stops uploading. The original file uploaded instantly [which I copied and renamed for the other tests].

Comment 11

[James Socol [:jsocol, :james]]

Still seeing:

====

  File "/data/www/master.support.mozilla.com/kitsune/vendor/src/django/django/core/files/storage.py", line 196, in exists
    return os.path.exists(self.path(name))

  File "/data/virtualenvs/BASELINE/lib64/python2.6/genericpath.py", line 18, in exists
    st = os.stat(path)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 83-84: ordinal not in range(128)

====

I think we may need to change the filename on upload. This exact thing works locally... It's too environmentally dependent to accept non-ascii filenames.

Comment 12

[Paul Craciunoiu [:paulc]]

What if we used slugify? It should take out non-ascii, and if the result is empty (or too short), we append/default to an integer timestamp or str(some date format).

Comment 13

[James Socol [:jsocol, :james]]

Why worry about branching, and checking the output of slugify? Let's just name img-timestamp.format or something.

Comment 14

[[:Cww]]

Would that break the search function in the media gallery? (I'm assuming not since you still have to give it a title.)

Comment 15

[James Socol [:jsocol, :james]]

(In reply to comment #14)
> Would that break the search function in the media gallery? (I'm assuming not
> since you still have to give it a title.)

We don't search on the filename in the gallery, only title and description.

Comment 16

[James Socol [:jsocol, :james]]

Cheating a little on my "let's close the P2s today" goal, but given the error rate for this--non-existent so far--I think there are much more important things to take care of for 2.3.

Comment 19

[Paul Craciunoiu [:paulc]]

It took quite some time, but this is finally closed. Landed on master:
https://github.com/jsocol/kitsune/commit/cdd18718f845d2547858ff1cfed569e7ce6cc43c

File names are now of this format:
YYYY-MM-DD-HH-MM-SS-123456.extension

Where 123456 is a substring of md5(original file name).


This only applies to newly uploaded files and not to existing files, obviously.

Comment 20

[Rebecca Billings [:rbillings]]

Verified able to upload with normal, high-ascii or high-unicode filenames.