1723190 - New SUMO article which looks perfect in preview unexpectedly gets truncated after submit due to extended UTF8 character like 🞇, discards article source for good

Status: RESOLVED FIXED

(support.mozilla.org :: Knowledge Base Software, defect)

Description

[Thomas D. (:thomas8)]

Attachment: Screenshot 1: Perfect Preview VS. Content truncation at special character 🞇 after Submit!

STR

1. Create a fine, long, new SUMO article via https://support.mozilla.org/en-US/kb/new
2. Use some UTF8 extended characters at the beginning of your article, like 🞇 as a correct representation of a checked radiobox in instructions.
3. Preview your long new article.
4. Submit your long new article.

Actual result

• In the article preview, long new article looks perfect from first to last line; special characters like 🞇 correctly shown.
• After submission, article gets truncated at the first extended UTF8 character like 🞇 - that's cunning!
• Mediawiki source code of the article also truncated, so most of your fine, long, new article content is GONE for GOOD. That's significant dataloss.
• Curse Knowledge Base Software for annihilating your work 🤬🤬🤬!!! (At least on BMO, these no longer cause truncation - can we learn from their upgrade experience?)

Here's an example of truncation: https://support.mozilla.org/en-US/kb/customize-date-time-formats-thunderbird/revision/226645

Expected result

• Update Knowledge Base Software/Database to accept extended UTF8 characters like 🞇.
• Consistency between article preview and actual article after submit.

Until then:

• As long as those special characters cannot be handled, stop making a fool of users by showing them a perfect preview, only to lose unpredictable chunks of their content upon submission, just because of a single wrong character.
• Warn/sanitize at preview time.
• Or sanitize before submission.
• Or at least put a big warning banner on https://support.mozilla.org/en-US/kb/new that special characters can cause dataloss after submission, and that it's always better to maintain a copy of your article source in an external editor as you work along and before submission.

It's surprising that such things still happen in 2021.

Full disclosure: I have not been affected by the dataloss this time, because I'm already aware that SUMO's new article form must never be trusted at composition time. Apart from this bug, one wrong click or page refresh will also erase all of your contents. So I'm composing articles in an external editor. Nevertheless, it's extremely annoying having to double-check each and every extended UTF8 character if it works or not - some others like ≡ work flawlessly.

Comment 1

[Leo McArdle [:leo]]

Thanks for the detailed report, this is a bizarre issue in MySQL where utf-8 isn't actually utf-8, but a proprietary subset which is incapable of storing 4-byte utf-8 characters.

Upgrading our database to actually support utf-8 is going to be quite an involved process, but in the meantime I've opened a PR to more elegantly handle 4-byte characters: https://github.com/mozilla/kitsune/pull/4867

In the worst case it'll just strip the offending character(s) out of the field, but in cases like KB articles it'll convert it to the relevant HTML numeric character reference instead, so it'll appear as intended in the parsed output.

Comment 2

[Thomas D. (:thomas8)]

Thanks :Leo, that's awesome!

BMO team might have some insights on Database migration, around Bug 1253535 - Fix emoji truncation by changing rowformat to dynamic, and convert all columns to utf8mb4.

Comment 3

[Kelimutu [:kiki]]

See also: https://github.com/mozilla/sumo/issues/435

Comment 4

[Ryan Johnson (:rjohnson)]

SUMO is running Postgres as of today, so this should be resolved.