1349316 - Include dotted forms from Latin Extended Additional unicode block for IDNA

Status: NEW

(Core :: Networking, enhancement, P3)

Description

[Manish Goregaokar [:manishearth]]

http://www.gṃail.com/


Extra dots like that are *very* hard to notice. https://en.wikipedia.org/wiki/Latin_Extended_Additional has a bunch of dotted forms which we should IDNA blacklist.

Comment 1

[Valentin Gosu [:valentin]]

I think this the right thing to do. Single dots are very difficult to notice.
Gerv, any opinion on this?

Comment 2

[Valentin Gosu [:valentin]]

Also, I am inclined to only blacklist the characters that have an extra dot, but not other features. For example, ệ (Latin Small Letter E with circumflex and dot below) seems different enough to not need blacklisting.

Comment 3

[Gervase Markham [:gerv]]

We tend not to take unilateral decisions on blacklisting characters (unless they spoof protocol elements such as : or /), but defer to UTS 39: http://unicode.org/reports/tr39/ . AFAICS, it allows these characters.

Do you have any idea which, if any, language orthographies use these characters?

Gerv

Comment 4

[VGR]

This is a pretty good reference to which languages use which diacritics (and other characters).

https://en.wikipedia.org/wiki/Wikipedia:Language_recognition_chart

Perhaps rather than blacklisting these characters using language recognition to determine whether to display punycode or the IDN.

IE: If TLD is .de and the domain contains umlauts display IDN, else display punycode. Another option perhaps is detecting the local language of the device and using that to determine whether to display punycode or IDN.

Comment 5

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 6

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 7

[Jungshik Shin]

FYI: Chrome dev (65.x) channel began to block the following characters:

[:IdentifierStatus=Allowed:] &  [:Ll:] &
[[\u01cd-\u01dc] [\u1c80-\u1c8f][\u1e00-\u1e9b]
 [\u1f00-\u1fff][\ua640-\ua69f][\ua720-\ua7ff]]

Note that U+1E9C - U+1EFF are left alone because they're used in Vietnamese. And, there are quite a lot of Latin letters with dot below in that range.

Comment 8

[Gervase Markham [:gerv]]

Jungshik: is Chrome now going down the path of picking ranges of IDNA-allowed characters and unilaterally deciding to disallow them? Do you have any criteria or controls on this process to help you decide what you can safely disallow and what not?

Gerv

Comment 9

[Jungshik Shin]

The set has 218 code points. A criterion was whether they're likely to be used for modern languages. (I didn't check them against the CLDR's exemplar character sets for all languages, but I guess there are only few, if any, characters in the intersection between the set I excluded and CLDR's exemplar character sets for all locales). 

The biggest block (Greek Extended) has 112 code points. Greek NIC does not allow them, either. I don't see a point of allowing them. (they're precomposed polytonic Greek letters. Modern Greek orthography does not use them). 

The second biggest block (Latin Ext. Additional) with 79 code points:  Gaelic (old orthography), Indic transliterations, Medievalist additions, etc.   ( http://www.unicode.org/charts/PDF/U1E00.pdf ). 

8 code points excluded from Latin Ext B are all for Pinyin. 

9 code points excluded from Latin Ext D are for Medievalist, Mayanists, and UPA. 

1 code point from Cyr Ext B is for Old Cyrillic. 

9 code points from Cyr Ext C are for Historic letter variants.

Comment 10

[Gervase Markham [:gerv]]

Jungshik: thank you for the detailed list of _what_ you have banned. As for _why_, you say "A criterion was whether they're likely to be used for modern languages". Is that the only criterion? If not, what are the other(s)? And how did you decide "likely"?

I really wanted to avoid simply starting to ban sets of otherwise-permitted characters, but if that ship has sailed, I do think we need:

0) Consistency across browsers (can you rope the others in for a discussion?)
1) Documentation as to exactly which characters have been added to the banned list, and why
2) Some sort of process for a language community to raise an objection and perhaps get a decision reversed

Do you think we can get all those things?

Gerv

Comment 12

[andreadari91]

I had forgotten this bug for a while, but today I was testing this on chromium and it has been fixed while in firefox is still showing https://www.airfraṇce.com/ on the address bar.

The funny thing is that firefox shows the punycode in the connection error message...:

Firefox can’t establish a connection to the server at www.xn--airfrace-n99c.com.

Valentin?

Comment 13

[andreadari91]

I add the reference to the chromium bug tracker for the same bug:

https://bugs.chromium.org/p/chromium/issues/detail?id=770709

Comment 14

[Valentin Gosu [:valentin]]

Attachment: Bug 1349316 - Include dotted forms from Latin Extended Additional unicode block for IDNA

Comment 15

[Valentin Gosu [:valentin]]

Don't have time to finish this right now.

Comment 16

[BugBot [:suhaib / :marco]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:kershaw, could you have a look please?

For more information, please visit BugBot documentation.