843689 - Eliminate the IDN TLD whitelist

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Daniel Veditz [:dveditz]]

Once the algorithm in bug 722299 lands we no longer need the tld-based IDN whitelist because the algorithm is an embodiment of the reasonable rules already agreed to by the registries. The registries only control the domains they issue, however, so the algorithm is an improvement that protects against customers who create sub-domains that do not follow the same anti-spoofing rules.

After landing 722299 we can wait a release or two before removing the whitelist, to make sure nothing breaks.

Comment 1

[Gervase Markham [:gerv]]

As dveditz said in the security review, it might be wise of us to contact the listed registries and warn them that this is our plan.

smontagu: how hard would it be to write a tiny C++ app which took a list of domains on stdin and spat out a list of ones which failed the check on stdout?

We could then offer this app to registries who wanted to check on the impact it would have on their TLD, and give us feedback.

Gerv

Comment 2

[Gervase Markham [:gerv]]

dveditz, smontagu: should we just go ahead and do this? No-one seems to be challenging UTR#39 (and so our algorithm) as being the basic right way to do this. And, as noted in bug 1164788, the whitelist means that sub-subdomains of whitelisted TLDs get a free pass. This might be exploitable for places like appspot.com and its friends (or ones not run as competently) because people can see someone with addons.examplecloudprovider.whitelistedtld and register a subdomain to spoof the "addons" part with e.g. Cyrillic. 

Gerv

Comment 3

[Simon Montagu :smontagu]

My chief concern is whether this would "break" (i.e. display in punycode) existing domains, which implies the question whether there are whitelisted domains whose registration policies are less restrictive than our display algorithm. Do we have any information on that?

Since the whitelist is pref-controlled, we can do this in the first instance by flipping the default value, and retreating will be easy if necessary.

Comment 4

[Daniel Veditz [:dveditz]]

> dveditz, smontagu: should we just go ahead and do this?

Of course I'm in favor, IMHO it's two years two late as it is.

Comment 5

[Gervase Markham [:gerv]]

smontagu: let's approach the compatibility issues by sending this change up to beta, and seeing if we get any bug reports. We can hold it back from release very easily - it's a single pref flip, right? (The use_whitelist pref.)

Gerv

Comment 6

[Simon Montagu :smontagu]

Attachment: Flip the pref

https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa438d4bcc34

Comment 7

[Simon Montagu :smontagu]

*sigh* Bug 1172802 reminded me that we probably need a fix for bug 858417 before checking this in. Here's an example of a live IDN that gets punycoded if we turn off the whitelist: http://nam-việt.vn/

Comment 8

[Gervase Markham [:gerv]]

Comment on attachment 8612202 [details] [diff] [review]
Flip the pref

Review of attachment 8612202 [details] [diff] [review]:
-----------------------------------------------------------------

r=gerv; but we need to monitor the compat impact of this, rather than just forgetting about it and letting it ride the trains.

Gerv

Comment 9

[Gervase Markham [:gerv]]

Dan: are you OK with us checking this in and keeping an eye on it for compat?

Gerv

Comment 10

[Daniel Veditz [:dveditz]]

If Simon is "away" who will be keeping an eye on it? Sounds like there are at least two known and unfixed compat bugs in comment 7. But sure, checking it in is an improvement in my book.

Comment 11

[Simon Montagu :smontagu]

(In reply to Simon Montagu away :smontagu from comment #7)
> *sigh* Bug 1172802 reminded me that we probably need a fix for bug 858417
> before checking this in. Here's an example of a live IDN that gets punycoded
> if we turn off the whitelist: http://nam-việt.vn/

Gerv pointed out that the Latin diacritics used in Vietnamese are all now "Allowed ; Recommended" in Unicode 8.0 so I will now check this in. (I am still "away" in the sense that I'm not working full-time on Mozilla code, but I am around enough to keep an eye open for regressions).

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/800c27046149

Comment 13

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/800c27046149