597070 - UnicodeEncodeError when saving output of wiki_to_html straight to the database

Status: VERIFIED FIXED

(support.mozilla.org :: General, defect, P1)

Description

[Paul Craciunoiu [:paulc]]

Getting this on migrating KB articles (re: bug 586487). Filing after a discussion with James -- it may be occuring on the search branch as well.

Stack trace:
Processing document 3502...
Traceback (most recent call last):
  File "./manage.py", line 49, in <module>
    execute_manager(settings)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/Users/paulc/Sites/kitsune/apps/wiki/management/commands/migrate_kb.py", line 187, in handle
    document = create_document(tiki_document)
  File "/Users/paulc/Sites/kitsune/apps/wiki/management/commands/migrate_kb.py", line 134, in create_document
    revision = create_revision(td, document, is_approved)
  File "/Users/paulc/Sites/kitsune/apps/wiki/management/commands/migrate_kb.py", line 113, in create_revision
    revision.save()
  File "/Users/paulc/Sites/kitsune/apps/wiki/models.py", line 161, in save
    self.document.save()
  File "/Users/paulc/Sites/kitsune/apps/wiki/models.py", line 103, in save
    super(Document, self).save(*args, **kwargs)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/db/models/base.py", line 434, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/db/models/base.py", line 500, in save_base
    rows = manager.using(using).filter(pk=pk_val)._update(values)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/db/models/query.py", line 491, in _update
    return query.get_compiler(self.db).execute_sql(None)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/db/models/sql/compiler.py", line 861, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/db/models/sql/compiler.py", line 727, in execute_sql
    cursor.execute(sql, params)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/db/backends/util.py", line 15, in execute
    return self.cursor.execute(sql, params)
  File "/Users/paulc/Sites/kitsune/vendor/src/django/django/db/backends/mysql/base.py", line 86, in execute
    return self.cursor.execute(query, args)
  File "/Users/paulc/.virtualenvs/kitsune/lib/python2.6/site-packages/MySQLdb/cursors.py", line 158, in execute
    query = query % db.literal(args)
  File "/Users/paulc/.virtualenvs/kitsune/lib/python2.6/site-packages/MySQLdb/connections.py", line 265, in literal
    return self.escape(o, self.encoders)
  File "/Users/paulc/.virtualenvs/kitsune/lib/python2.6/site-packages/MySQLdb/connections.py", line 198, in string_literal
    return db.string_literal(obj)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 32-33: ordinal not in range(128)

Comment 1

[Paul Craciunoiu [:paulc]]

Looks like this occurs because wiki_to_html doesn't return unicode. It returns a Markup object, which MySQLdb chokes on.

Comment 2

[Paul Craciunoiu [:paulc]]

http://github.com/jsocol/kitsune/commit/d2958f441c796c67979a99c0303960572a7bf121

Comment 3

[Rebecca Billings [:rbillings]]

Please add verification steps or [qa-], thanks

Comment 4

[James Socol [:jsocol, :james]]

Best test I can think of is approving a new revision. I think that's the only place we render wiki text to HTML and stick it in the DB.

Comment 5

[James Socol [:jsocol, :james]]

To clarify: approving a revision should work, not cause an ISE.

Comment 6

[Rebecca Billings [:rbillings]]

Verified approving a revision works well