1193265 - 500 error when submitting an app containing some specific unicode characters

Status: RESOLVED WONTFIX

(Marketplace Graveyard :: Code Quality, defect, P4)

Description

[Mathieu Pillard [:mat]]

STR:
- Make an app using U+1D306 TETRAGRAM FOR CENTRE (

Comment 1

[Mathieu Pillard [:mat]]

Amusingly, Bugzilla does suffer from this bug and ate my STR :(
STR:
- Make an app using U+1D306 TETRAGRAM FOR CENTRE unicode symbol in the name
- Submit it to Marketplace

Expected results:
- Validation passes, step 2 of the submission process shows you the name correctly

Actual results:
- Validation passes, but step 2 of the submission process shows "Oops! We had an error.".



Sentry: http://sentry.dmz.phx1.mozilla.com/marketplace-dev/marketplace-dev/group/28732/

Actual error comes from MySQL:
OperationalError: (1366, "Incorrect string value: '\\xF0\\x9D\\x8C\\x86 B...' for column 'localized_string' at row 1")

This is because in MySQL, the "utf8" charset sucks. The fix is to use "utf8mb4" charset instead, introduced in Mysql 5.5.3 version. This article details the issue and suggests a way to migrate existing data to the new charset: https://mathiasbynens.be/notes/mysql-utf8mb4

Note: we're in strict mode, so we avoid the security issue, MySQL is returning an error instead of just a warning.

Comment 3

[David Durst [:ddurst]]

While in strict mode, this is not a security issue. But it's worth noting that if this codebase goes to another team, that *could* get changed. But changing the collation and migrating the data would be largely on Ops.

Jason -- we were going to WONTFIX this, but I thought I'd get your opinion first. I'd hate for this to go off the radar and then have some future person change it and not know about this.

Comment 4

[Jason Thomas [:jason]]

This would require downtime to do the migration over to utf8mb4. Ideally we would setup a new database server with the new mysql settings and the new schema and then import the data. This would be similar to the maintenance we performed to move from migrations schema to django generated schema.

We would also need to investigate the maximum length of index keys and columns (which is mentioned in the article in comment 1).

Comment 5

[David Durst [:ddurst]]

With the future of Marketplace on the bubble, we're not going to invest in ops upgrades without cause. Assume current state until otherwise indicated.