320568 - Need to set up an IDN testing environment

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations: Projects, task)

Description

[Neil Harris]

We need to set up a DNS test environment to provide actual DNS lookups for a variety of IDN scenarios, including testing malformed and/or exotic IDNs, and the possibility of fully-internationalized TLDs.

A suitable configuration for this would be 
1) a zone file for a fake rootserver
2) a webserver, programmed to respond to any domain name
3) a collection of links in an HTML page, which contain various testcase IDNs

Best of all would be if we could have:

4) an automated regression tester that can visit these URLs, testing for appropriate behaviour

Comment 1

[Bob Clary [:bc] (inactive)]

(In reply to comment #0)
> 
> A suitable configuration for this would be 
> 1) a zone file for a fake rootserver

not sure how to do this.

> 2) a webserver, programmed to respond to any domain name

not sure how to do this in general.

> 3) a collection of links in an HTML page, which contain various testcase IDNs

if this is all that is needed, it would be easy to set up a host file with the appropriate machine names all mapped to a virtual server somewhere.

> 
> Best of all would be if we could have:
> 
> 4) an automated regression tester that can visit these URLs, testing for
> appropriate behaviour

The automated tests I use based on Makefiles and Spider would be good for this. The question is what is the "appropriate behavior".

If someone can come up with the list in 4) along with the expected behavior, I can easily set this up to run on the QA machines. If you want this to be generally accessible, someone needs to approve a location in CVS.

Comment 2

[Dave Miller [:justdave]]

(In reply to comment #1)
> (In reply to comment #0)
> > A suitable configuration for this would be 
> > 1) a zone file for a fake rootserver
> 
> not sure how to do this.

This should be pretty easy to set up.  Let me know when and I can give you a hand with it.

> > 2) a webserver, programmed to respond to any domain name
> 
> not sure how to do this in general.

Apache does this by default.  Any domain name that doesn't match a defined vhost gets the first-defined vhost in the config.  If you want different content on different domains, that's when the config starts getting tedious :)

Comment 3

[Dave Miller [:justdave]]

-> default assignee

Though from discussion, sounds like Bob might kinda sorta own this. Feel free to take it.

Comment 4

[Bob Clary [:bc] (inactive)]

(In reply to comment #2)
> 
> Apache does this by default.  Any domain name that doesn't match a defined
> vhost gets the first-defined vhost in the config.  If you want different
> content on different domains, that's when the config starts getting tedious :)

I assume this depends on 1) above.

If we have a limited set of hostnames, I would think a hosts file would be easier. Wouldn't using a zone file and fake root server be machine wide and preclude other uses of the machine?

Comment 5

[Dave Miller [:justdave]]

(In reply to comment #4)
> I assume this depends on 1) above.

Of course. The domain needs to actually get to the webserver. :)

> If we have a limited set of hostnames, I would think a hosts file would be
> easier. Wouldn't using a zone file and fake root server be machine wide and
> preclude other uses of the machine?

Yes, setting up a fake root and pointing your machine at it would quite likely make any other use of that machine somewhat useless until you changed it back to point at a real DNS server.  If there's a limited set of domains, then a hosts file on the client machine doing the testing is probably the best bet.

Comment 6

[Neil Harris]

Attachment: extra /etc/hosts entries for IDN testing

Add these entries to /etc/hosts to create a simple IDN testing environment: note that no actual DNS lookups will occur on the wire, so this is not a proper full test of IDN

Comment 7

[Neil Harris]

Attachment: HTML file for testing above IDNs in UTF-8 charset

Note that this document has a charset of UTF-8, and the raw URLs and raw text in in it are encoded in UTF-8.

Comment 8

[Neil Harris]

Attachment: HTML file for testing above IDNs in Latin-1 charset

This has only the subset of the above names that can be encoded entirely in iso-8859-1. The raw text and raw links are coded in iso-8859-1. 

The percent-encoded iso-8859-1 forms _should not work_, if I read RFC 3986 correctly.

RFC 3986 says:

  When a new URI scheme defines a component that represents textual
   data consisting of characters from the Universal Character Set [UCS],
   the data should first be encoded as octets according to the UTF-8
   character encoding [STD63]; then only those octets that do not
   correspond to characters in the unreserved set should be percent-
   encoded.  For example, the character A would be represented as "A",
   the character LATIN CAPITAL LETTER A WITH GRAVE would be represented
   as "%C3%80", and the character KATAKANA LETTER A would be represented
   as "%E3%82%A2".

See also bug 309671 for the percent-encoding issue.

Comment 9

[Neil Harris]

Note that to be able to see the non-ASCII characters in the URL bar and status bar, you _should_ have to create and set the following about:config boolean variable:

network.IDN.whitelist.idntest true

Comment 10

[Neil Harris]

Note what happens when hovering the mouse over the percent-encoded links in the respective test HTML files.

Comment 11

[Neil Harris]

Attachment: Small python CGI program for announcing IDN name of web host

This small Python CGI program (warning: only tested on the command line, not on a web server) should report both the internationalized and raw name of the web server it has been called on, hopefully using the correct DNS name used in the URL.

Comment 12

[Neil Harris]

Attachment: Zone file fragment for the above: please adapt as necessary

If this zone file was used for the "idntest.mozilla.org" domain, with the IP addresses changed from 10.0.0.1 to point at an appropriately configured web server, this (with suitably edited versions of the HTML files above) would be all that was needed to test IDNs in a real, live, environment, with real DNS bytes going over the wire, and sniffable using Ethereal.

Better, if a proper host is provided, why not use a DNS wildcard, and avoid any need for reconfiguration when new names are needed?

Note that this needs to be in .org space, not .com, since .org is by default IDN-whitelisted in production Mozilla products.

Comment 13

[Reed Loden [:reed]]

Is this still needed?

Comment 14

[Jeff Walden [:Waldo]]

It may be needed for some of the underlying DNS functionality, but it's not needed for anything that relies on IDN as Mochitest can, and does, emulate any arbitrary domains it needs.

Comment 15

[Jeremy Orem [:oremj]]

Changing QA Contact.

Comment 16

[matthew zeier [:mrz]]

From comment #14, it sounds like it's not needed.  Re-open if I'm wrong.