Need to set up an IDN testing environment

RESOLVED WONTFIX

Status

mozilla.org Graveyard
Server Operations: Projects
RESOLVED WONTFIX
12 years ago
3 years ago

People

(Reporter: Neil Harris, Unassigned)

Tracking

Details

Attachments

(5 attachments)

(Reporter)

Description

12 years ago
We need to set up a DNS test environment to provide actual DNS lookups for a variety of IDN scenarios, including testing malformed and/or exotic IDNs, and the possibility of fully-internationalized TLDs.

A suitable configuration for this would be 
1) a zone file for a fake rootserver
2) a webserver, programmed to respond to any domain name
3) a collection of links in an HTML page, which contain various testcase IDNs

Best of all would be if we could have:

4) an automated regression tester that can visit these URLs, testing for appropriate behaviour
Assignee: darin → server-ops
Component: Networking → Server Operations
Product: Core → mozilla.org
QA Contact: benc → justin
Version: Trunk → other
(Reporter)

Updated

12 years ago
Blocks: 316444, 316727, 318006, 318112
No longer blocks: 316730

Updated

12 years ago
Component: Server Operations → Server Operations Projects

Comment 1

12 years ago
(In reply to comment #0)
> 
> A suitable configuration for this would be 
> 1) a zone file for a fake rootserver

not sure how to do this.

> 2) a webserver, programmed to respond to any domain name

not sure how to do this in general.

> 3) a collection of links in an HTML page, which contain various testcase IDNs

if this is all that is needed, it would be easy to set up a host file with the appropriate machine names all mapped to a virtual server somewhere.

> 
> Best of all would be if we could have:
> 
> 4) an automated regression tester that can visit these URLs, testing for
> appropriate behaviour

The automated tests I use based on Makefiles and Spider would be good for this. The question is what is the "appropriate behavior".

If someone can come up with the list in 4) along with the expected behavior, I can easily set this up to run on the QA machines. If you want this to be generally accessible, someone needs to approve a location in CVS.
(In reply to comment #1)
> (In reply to comment #0)
> > A suitable configuration for this would be 
> > 1) a zone file for a fake rootserver
> 
> not sure how to do this.

This should be pretty easy to set up.  Let me know when and I can give you a hand with it.

> > 2) a webserver, programmed to respond to any domain name
> 
> not sure how to do this in general.

Apache does this by default.  Any domain name that doesn't match a defined vhost gets the first-defined vhost in the config.  If you want different content on different domains, that's when the config starts getting tedious :)
-> default assignee

Though from discussion, sounds like Bob might kinda sorta own this. Feel free to take it.
Assignee: server-ops → nobody

Comment 4

12 years ago
(In reply to comment #2)
> 
> Apache does this by default.  Any domain name that doesn't match a defined
> vhost gets the first-defined vhost in the config.  If you want different
> content on different domains, that's when the config starts getting tedious :)

I assume this depends on 1) above.

If we have a limited set of hostnames, I would think a hosts file would be easier. Wouldn't using a zone file and fake root server be machine wide and preclude other uses of the machine?
(In reply to comment #4)
> I assume this depends on 1) above.

Of course. The domain needs to actually get to the webserver. :)

> If we have a limited set of hostnames, I would think a hosts file would be
> easier. Wouldn't using a zone file and fake root server be machine wide and
> preclude other uses of the machine?

Yes, setting up a fake root and pointing your machine at it would quite likely make any other use of that machine somewhat useless until you changed it back to point at a real DNS server.  If there's a limited set of domains, then a hosts file on the client machine doing the testing is probably the best bet.
(Reporter)

Comment 6

12 years ago
Created attachment 207933 [details]
extra /etc/hosts entries for IDN testing

Add these entries to /etc/hosts to create a simple IDN testing environment: note that no actual DNS lookups will occur on the wire, so this is not a proper full test of IDN
(Reporter)

Comment 7

12 years ago
Created attachment 207936 [details]
HTML file for testing above IDNs in UTF-8 charset

Note that this document has a charset of UTF-8, and the raw URLs and raw text in in it are encoded in UTF-8.
(Reporter)

Comment 8

12 years ago
Created attachment 207938 [details]
HTML file for testing above IDNs in Latin-1 charset

This has only the subset of the above names that can be encoded entirely in iso-8859-1. The raw text and raw links are coded in iso-8859-1. 

The percent-encoded iso-8859-1 forms _should not work_, if I read RFC 3986 correctly.

RFC 3986 says:

  When a new URI scheme defines a component that represents textual
   data consisting of characters from the Universal Character Set [UCS],
   the data should first be encoded as octets according to the UTF-8
   character encoding [STD63]; then only those octets that do not
   correspond to characters in the unreserved set should be percent-
   encoded.  For example, the character A would be represented as "A",
   the character LATIN CAPITAL LETTER A WITH GRAVE would be represented
   as "%C3%80", and the character KATAKANA LETTER A would be represented
   as "%E3%82%A2".

See also bug 309671 for the percent-encoding issue.
(Reporter)

Comment 9

12 years ago
Note that to be able to see the non-ASCII characters in the URL bar and status bar, you _should_ have to create and set the following about:config boolean variable:

network.IDN.whitelist.idntest true

Attachment #207936 - Attachment mime type: text/html → text/html; charset=utf-8
Attachment #207938 - Attachment mime type: text/html → text/html; charset=ISO-8859-1
(Reporter)

Comment 10

12 years ago
Note what happens when hovering the mouse over the percent-encoded links in the respective test HTML files.
(Reporter)

Comment 11

12 years ago
Created attachment 207946 [details]
Small python CGI program for announcing IDN name of web host

This small Python CGI program (warning: only tested on the command line, not on a web server) should report both the internationalized and raw name of the web server it has been called on, hopefully using the correct DNS name used in the URL.
(Reporter)

Comment 12

12 years ago
Created attachment 207950 [details]
Zone file fragment for the above: please adapt as necessary

If this zone file was used for the "idntest.mozilla.org" domain, with the IP addresses changed from 10.0.0.1 to point at an appropriately configured web server, this (with suitably edited versions of the HTML files above) would be all that was needed to test IDNs in a real, live, environment, with real DNS bytes going over the wire, and sniffable using Ethereal.

Better, if a proper host is provided, why not use a DNS wildcard, and avoid any need for reconfiguration when new names are needed?

Note that this needs to be in .org space, not .com, since .org is by default IDN-whitelisted in production Mozilla products.
Is this still needed?
It may be needed for some of the underlying DNS functionality, but it's not needed for anything that relies on IDN as Mochitest can, and does, emulate any arbitrary domains it needs.
Changing QA Contact.
QA Contact: justin → mrz
From comment #14, it sounds like it's not needed.  Re-open if I'm wrong.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → WONTFIX
(Assignee)

Updated

3 years ago
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.