733341 - Building XPI removes non ASCII characters from description

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Add-on Builder, defect, P1)

Description

[Piotr Zalewa [:zalun]]

Copying from a bug where this description doesn't belong (bug 732602 "-ons built using Builder don't work if there are special chars in the description")

--------
As mentioned in the bug title, this also happens when using special chars, such as German umlauts (ä, ö, ü, ß).

When putting these chars into the Description, they're shown correctly when re-opening properties. Once you reload the Builder, however, they're gone. They're also gone when downloading the XPI.

Maybe a general move to UTF8 for the description could solve that.
---------

The reason I moved it here is that Firefox installs XPIs build like that.
I also confirm non ASCII characters are removed.

Comment 1

[Piotr Zalewa [:zalun]]

Description in plain ASCII is currently a feature, not a bug.
It is sanitized in here: https://github.com/mozilla/FlightDeck/blob/master/apps/jetpack/models.py#L1956

I'm leaving this bug open, as I agree we should allow UTF8 characters in description and only prevent certain characters to be placed.

Comment 2

[Daniel Buchner [:dbuc]]

I thought we had converted *every* part of stored data to UTF8? This is causing XPI bustage and needs to be fixed asap. Can we have this for tomorrow?

Comment 3

[Piotr Zalewa [:zalun]]

This commit fixes the issue, please review: 
https://github.com/zalun/FlightDeck/commit/c341a037e79c1c557ba71b118cd77f0bb0a23e19
If it will achieve r+ and will work fine in -dev it can go live tomorrow for sure.

@Daniel

There was nothing wrong with XPI created were the utf8 characters were stripped from description.

no - we can't allow every bit of data to be utf-8

Another field which is forced to be ASCII only is the ``full_name`` - would you like it to be?
This might lead to an issue, as the ``name``, which is created from it is a file name and AFAIR it should be ASCII, @clouserw, please correct me if I'm wrong.
Then there is ``version_name`` which should be plain ASCII as well.

I think we should have a table of all type of data and check how these are used in AMO

Comment 4

[Daniel Buchner [:dbuc]]

There were reports of certain data entered into the name and description field that were borking add-on creation on Builder, but when the same data was used directly with the XPI to generate an add-on, it produced a working add-on just fine. Such a bork suggests to me that there is an encoding issue, is there something else that is causing us not be on par with the SDK's XPI generation capability with identical data sets?

Comment 5

[Daniel Buchner [:dbuc]]

Correction: "when the same data was used directly with the *SDK* to generate an add-on"

Comment 6

[Piotr Zalewa [:zalun]]

Yes, this is a different bug though. The other one is mentioned in the description of this bug.