Closed Bug 968064 Opened 10 years ago Closed 10 years ago

Effective Public Suffix List on mxr missing new gTLDs - vs PublicSuffix.org version being up-to-date

Categories

(Core Graveyard :: Networking: Domain Lists, defect)

Product:
Core Graveyard
Component:
Networking: Domain Lists
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: noloader, Unassigned)

Details

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20131030 Firefox/17.0 Iceweasel/17.0.10 (Nightly/Aurora)
Build ID: 20131030041028

Steps to reproduce:

Viewed Mozilla's list of Effective Top Level Domains at http://publicsuffix.org/list/ and http://publicsuffix.org/list/effective_tld_names.dat.

And viewed ICANN's new list of gTLDs at http://newgtlds.icann.org/en/program-status/delegated-strings.


Actual results:

ICANN's new gTLD's appear to be missing from Mozilla's Effective Top Level Domains. For example, .VOTING and .TOKYO were missing.


Expected results:

The new gTLDs should have been listed in Mozilla's list.

**********

This does not appear to be Bug 860679.

This may be Bug 962360, but I cannot tell (there was no useful commentary, so its hard to say what the complaint was).

**********

Here's the column of interest from ICANN's page.

.VOTING
.TOKYO
.NAGOYA
.DATING
.COMMUNITY
.BOUTIQUE
.BARGAINS
.TIENDA
.WORKS
.WED
.EXPERT
.WATCH
.KIM
.COOL
.TOOLS
.CLUB
.BUILD
.PICS
.PINK
.LUXURY
.PHOTO
.GIFT
.网络 (xn--io0a7i) – Chinese for "network"
.公司 (xn--55qx5d) – Chinese for "company"
.MONASH
.RICH
.RED
.SHIKSHA
.中信 (xn--fiq64b) – Chinese for "CITIC"
.LINK
.GUITARS
.ZONE
.CHEAP
.MARKETING
.DEMOCRAT
.SOCIAL
.AGENCY
.MODA
.DANCE
.BERLIN
集团 (xn--3bst00m) – Chinese for "group"
我爱你 (xn--6qq986b3xl) – Chinese for "I love you"
中文网 (xn--fiq228c5hs) – Chinese for "Chinese network"
.WANG
.KIWI
.WIEN
.EMAIL
.IMMOBILIEN
.在线 (xn--3ds443g) – Chinese for "online"
.INSTITUTE
.CEO
.SOLAR
.FARM
.EDUCATION
.GLASS
.ONL
.INTERNATIONAL
.CODES
.TRAINING
.HOUSE
.KAUFEN
.NINJA
.REPAIR
.BUILDERS
.COFFEE
.FLORIST
.HOLIDAY
.SOLUTIONS
.BUZZ
.SUPPORT
.RECIPES
.COMPUTER
.ACADEMY
.CAREERS
.CAB
公益 (xn--55qw42g) – Chinese for "charity" 
政务 (xn--zfr164b) – Chinese for "government" 
.SYSTEMS
.DOMAINS
.VIAJES
.COMPANY
.CAMP
.LIMO
.MANAGEMENT
.PHOTOS
.SHOES
.CENTER
.RUHR
.MENU
.UNO
みんな (xn--q9jyb4c) - Japanese for "everyone"
.diamonds
.tips
.photography
.directory
.enterprises
.kitchen
.today
.plumbing
.graphics
.contractors
.gallery
.sexy
.construction
.tattoo
.technology
.estate
.land
.bike
.VENTURES
.CAMERA
.CLOTHING
.LIGHTING
.SINGLES
.VOYAGE
.GURU
.HOLDINGS
.EQUIPMENT
شبكة (xn--ngbc5azd) – Arabic for "web/network"
онлайн (xn--80asehdb) – Russian for "online"
сайт (xn--80aswg) – Russian for "site"
游戏(xn--unup4y) – Chinese for "game(s)"
Summary: Effeective Public Suffix List missing new gTLDs → Effective Public Suffix List missing new gTLDs
Why is this bug hidden? Is this a security issue?
Flags: needinfo?(noloader)
the other bugs are public and I can see no reason why this would be sec sensitive, unhiding
Group: core-security
In attempting to validate this report, the strings reported to be missing are present within the PSL.

The community volunteers who are updating the PSL are tracking the contract announcements from ICANN, which precede the root delegation of new TLDs, and are updating 1-2 times per month from that information.

Was this problem report based upon http://publicsuffix.org/list/effective_tld_names.dat or perhaps a cached or older version, or was the search perhaps case-sensitive where it may have missed a lower-cased version of an upper case string (or vice-versa)?

Recommend closing this as it seems to not be possible to reproduce.

(In reply to Jeffrey Walton from comment #0)
> User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20131030
> Firefox/17.0 Iceweasel/17.0.10 (Nightly/Aurora)
> Build ID: 20131030041028
> 
> Steps to reproduce:
> 
> Viewed Mozilla's list of Effective Top Level Domains at
> http://publicsuffix.org/list/ and
> http://publicsuffix.org/list/effective_tld_names.dat.
> 
> And viewed ICANN's new list of gTLDs at
> http://newgtlds.icann.org/en/program-status/delegated-strings.
> 
> 
> Actual results:
> 
> ICANN's new gTLD's appear to be missing from Mozilla's Effective Top Level
> Domains. For example, .VOTING and .TOKYO were missing.
> 
> 
> Expected results:
> 
> The new gTLDs should have been listed in Mozilla's list.
> 
> **********
> 
> This does not appear to be Bug 860679.
> 
> This may be Bug 962360, but I cannot tell (there was no useful commentary,
> so its hard to say what the complaint was).
> 
> **********
> 
> Here's the column of interest from ICANN's page.
> 
> .VOTING
> .TOKYO
> .NAGOYA
> .DATING
> .COMMUNITY
> .BOUTIQUE
> .BARGAINS
> .TIENDA
> .WORKS
> .WED
> .EXPERT
> .WATCH
> .KIM
> .COOL
> .TOOLS
> .CLUB
> .BUILD
> .PICS
> .PINK
> .LUXURY
> .PHOTO
> .GIFT
> .网络 (xn--io0a7i) – Chinese for "network"
> .公司 (xn--55qx5d) – Chinese for "company"
> .MONASH
> .RICH
> .RED
> .SHIKSHA
> .中信 (xn--fiq64b) – Chinese for "CITIC"
> .LINK
> .GUITARS
> .ZONE
> .CHEAP
> .MARKETING
> .DEMOCRAT
> .SOCIAL
> .AGENCY
> .MODA
> .DANCE
> .BERLIN
> 集团 (xn--3bst00m) – Chinese for "group"
> 我爱你 (xn--6qq986b3xl) – Chinese for "I love you"
> 中文网 (xn--fiq228c5hs) – Chinese for "Chinese network"
> .WANG
> .KIWI
> .WIEN
> .EMAIL
> .IMMOBILIEN
> .在线 (xn--3ds443g) – Chinese for "online"
> .INSTITUTE
> .CEO
> .SOLAR
> .FARM
> .EDUCATION
> .GLASS
> .ONL
> .INTERNATIONAL
> .CODES
> .TRAINING
> .HOUSE
> .KAUFEN
> .NINJA
> .REPAIR
> .BUILDERS
> .COFFEE
> .FLORIST
> .HOLIDAY
> .SOLUTIONS
> .BUZZ
> .SUPPORT
> .RECIPES
> .COMPUTER
> .ACADEMY
> .CAREERS
> .CAB
> 公益 (xn--55qw42g) – Chinese for "charity" 
> 政务 (xn--zfr164b) – Chinese for "government" 
> .SYSTEMS
> .DOMAINS
> .VIAJES
> .COMPANY
> .CAMP
> .LIMO
> .MANAGEMENT
> .PHOTOS
> .SHOES
> .CENTER
> .RUHR
> .MENU
> .UNO
> みんな (xn--q9jyb4c) - Japanese for "everyone"
> .diamonds
> .tips
> .photography
> .directory
> .enterprises
> .kitchen
> .today
> .plumbing
> .graphics
> .contractors
> .gallery
> .sexy
> .construction
> .tattoo
> .technology
> .estate
> .land
> .bike
> .VENTURES
> .CAMERA
> .CLOTHING
> .LIGHTING
> .SINGLES
> .VOYAGE
> .GURU
> .HOLDINGS
> .EQUIPMENT
> شبكة (xn--ngbc5azd) – Arabic for "web/network"
> онлайн (xn--80asehdb) – Russian for "online"
> сайт (xn--80aswg) – Russian for "site"
> 游戏(xn--unup4y) – Chinese for "game(s)"
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> Why is this bug hidden? Is this a security issue?

Yes, its security related. I don't believe its a 0-day. But its difficult to discern if any CAs have issued a certifcate for *.VOTING and *.TOKYO because they are not required to publish the information.

What info do you need to satisfy the "needs info"?(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> Why is this bug hidden? Is this a security issue?
(In reply to Jothan Frakes from comment #3)
> In attempting to validate this report, the strings reported to be missing
> are present within the PSL.
> 
> The community volunteers who are updating the PSL are tracking the contract
> announcements from ICANN, which precede the root delegation of new TLDs, and
> are updating 1-2 times per month from that information.
> 
> Was this problem report based upon
> http://publicsuffix.org/list/effective_tld_names.dat or perhaps a cached or
> older version,
Yes.

> or was the search perhaps case-sensitive where it may have
> missed a lower-cased version of an upper case string (or vice-versa)?
No.

Forgive my ignorance here. Are the lists published at publicsuffix.org expected to be stale or out of date? If so, where can one get a current list?

I ask because I use `wget` to fetch the list and build it into my program for hostname name validation. My script actually uses http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_names.dat?raw=1 to avoid the HTML markup.

http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_names.dat and http://publicsuffix.org/list/effective_tld_names.dat appear to be equivalent. Both are missing, for example, .VOTING and .TOKYO.

> Recommend closing this as it seems to not be possible to reproduce.
I coud be wrong, but this seems very easy to reproduce.

Perhaps I'm using the wrong public suffix list?
(In reply to Jeffrey Walton from comment #5)
My bad... I should clarify this.

> > 
> > Was this problem report based upon
> > http://publicsuffix.org/list/effective_tld_names.dat
Yes.

> > or perhaps a cached or older version,
I don't believe so. I believe I am using the latest list published by Mozilla.
http://publicsuffix.org/list/effective_tld_names.dat?raw=1 and http://publicsuffix.org/list/effective_tld_names.dat

Please attach a copy of what you download so that we can review it.

Are you sure you're sure that you're not seeing these in the list you download?

Scratching my head a bit on this report, as 'it works for me' (and I have confirmed this from other parties).

Both of these versions of the list include the names that you're saying are absent...   but are not preceded by a dot, is that perhaps the issue?

Is it possible you're not parsing the entire file?

Due to the velocity of strings involved in the newTLD introduction under way with ICANN, these are not contained within the alphabetically ordered list, there's a section with ICANN new TLDs that get added in chronological order as the contracting announcements are transmitted by ICANN. 

Once the strings from this release phase (12 months - ish) are all or mostly delegated, the community editors will go back and arrange them into the alphabetically organized list, but because the list is managed by diff at the moment, this was the cleanest manner to keep up with the pace while maintaining a clean log of the changes.
I dug deeper and believe I have found what you are describing as an issue.

http://publicsuffix.org/list/effective_tld_names.dat  THIS IS UP TO DATE

Versions under http://mxr.mozilla.org/[whichever tree]/source/netwerk/dns/effective_tld_names.dat trail behind.

Solution for your specific purposes would be to use the version at publicsuffix.org in lieu of the others.
Oh ****...(In reply to Jothan Frakes from comment #8)
> I dug deeper and believe I have found what you are describing as an issue.
> 
> http://publicsuffix.org/list/effective_tld_names.dat  THIS IS UP TO DATE
> 
> Versions under http://mxr.mozilla.org/[whichever
> tree]/source/netwerk/dns/effective_tld_names.dat trail behind.
Oh ****....

Please close this report. Sorry for the false alarm.
Flags: needinfo?(noloader)
OK.  No worries, and also no need to apologize on this.  I don't think any of us mind taking the time to make sure things are working as expected for this important initiative, and bug reports are an important part of this.

Glad this is sorted for your needs... 

I did escalate this to investigate why the mxr tree is trailing, as I suspect that other 'consumers' of the PSL are using those versions from mxr in the mozilla-central tree or other versions of it as authority vs. the one at publicsuffix.org

It would be good to perhaps annotate a sole authoritative location for the most up to date version of the file to be retrieved from so others don't hit their head on this wall.

-j
Summary: Effective Public Suffix List missing new gTLDs → Effective Public Suffix List on mxr missing new gTLDs - vs PublicSuffix.org version being up-to-date
http://publicsuffix.org/list/effective_tld_names.dat is now the canonical URL for the PSL; it is updated from the mozilla-central copy once a day (I believe), and hosted on a CDN to better deal with (unwise) people who seem to think it needs downloading once a minute.

If I download
http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_names.dat
and
http://publicsuffix.org/list/effective_tld_names.dat
right now, there is no difference between them according to "diff". It's possible that MXR may sometimes have a lag, but it should not be significant. If you see one again, file a bug against MXR.

Gerv
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Resolution: --- → INVALID
I'm going to try an sneak this one in while effTLDs are being manipulated at the moment. Please let me know if you want another bug report (it kind of seems like too much extra work considering all the work that has been put in so far).

According to [1], ".dns" is now a meta-tld.

[1] "First public DNSChain server went online yesterday!", http://lists.randombit.net/pipermail/cryptography/2014-February/006237.html.
Jeffrey: each change should be a new bug report. But we only add TLDs from the ICANN root, and ".dns" is not one such, even if some group is using it privately.

Gerv
(In reply to Gervase Markham [:gerv] from comment #11)
> http://publicsuffix.org/list/effective_tld_names.dat is now the canonical
> URL for the PSL; it is updated from the mozilla-central copy once a day (I
> believe), and hosted on a CDN to better deal with (unwise) people who seem
> to think it needs downloading once a minute.

If this is the canonical URL, it should be provided under HTTPS, or it should have a detached, up-to-date cryptographic signature to protect the integrity of the traffic against an adversary in control of the network.

I'm not going to fetch it once a minute or anything (more like once a month or so to see if i need to update debian's publicsuffix package), but i care about the integrity of this data (and i would be really bummed to learn i'd shipped tampered data in debian).
Daniel: bug 978733.

Gerv
Gerv: Sorry, i don't appear to be authorized to access that bug, so i don't know what you're referring to.  If you have a way to give me access, i'd be happy to take a look.
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.