464591 - Percent-encoded host name can bypass IDN Normalization.

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[Masahiro YAMADA]

UTF-8 encoded characters can causes domain name spoofing on location bar and status bar.

testcase URL is looks like 
"http://www.example.com/www.mozilla.com/www.mozilla.org"
But, it contains three UTF-8 encoded non-ascii characters.
%E2%88%95 = U+2215 ('∕') : division slash
%D0%B0 = U+0430 ('а') : Cyrillic alphabet "a"
%E2%81%84 = U+2044 ('⁄') : fraction slash

Comment 1

[Masahiro YAMADA]

reproduced at

Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081111 Minefield/3.1b2pre

And, not reproduced on error message.

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

U+2215 and U+2044 are in the default value of the "network.IDN.blacklist_chars" preference:
http://hg.mozilla.org/mozilla-central/annotate/5ff478afc3f4/modules/libpref/src/init/all.js#l717
I would have hoped that would have prevented this from resolving.  But it seems like it's only intended to affect the display; however, that doesn't seem to be working.  Are we only blacklisting them if they're inside the TLD+1?

I'd sort of have hoped that the core IDN algorithms would either forbid these or normalize these to standard slashes (and then, in turn, forbid them because they're slashes).

Comment 3

[Masahiro YAMADA]

When Non-Ascii characters are inserted as HTML character reference , this issue is not reproduced.
for example: <a href="http://www.example.com&#x2215;www.mozill&#x0430;.com&#x2044;www.mozilla.org/">example</a>

Comment 4

[Masahiro YAMADA]

FYI: Other browsers test result:
Opera 9.62 for Windows: not reproduced. Opera reports it as illegal URL to user.
Safari: 3.1.2 for Windows reproduced at all displayed URL string.
   It may be needed to report this issue to Apple.

IE7: not reproduced (IE7 does not decode %-encoded string).

Comment 5

[Gervase Markham [:gerv]]

I don't know my way around this code :-| The list only seems to be used in this file:
http://mxr.mozilla.org/mozilla/source/netwerk/dns/src/nsIDNService.cpp
Neil used to be our guy, although I haven't heard from him in a while. CCing him :-)

Gerv

Comment 6

[Neil Harris]

Attachment: Bad percent-encoded URL testcase, in three different encodings

This HTML file encodes the same bad URL in three different encodings:
* percent-encoded, as in the reported test case
* UTF-8 encoded, in the native encoding of this HTML 4.01 document
* HTML numeric entity-encoded

Comment 7

[Neil Harris]

Here are the results of checking the above in two different versions of Firefox:

In Firefox 2.0.0.18:
* percent-encoded version displays misleadingly in status bar, displays percent-encoded in URL bar
* UTF8-encoded version displays Punycoded in both status bar and URL bar
* entity-encoded version displays Punycoded in both status bar and URL bar

In Firefox 3.0.4:
* percent-encoded version displays misleadingly in both status bar and URL bar
* UTF8-encoded version displays Punycoded in both status bar and URL bar
* entity-encoded version displays Punycoded in both status bar and URL bar

Comment 8

[Neil Harris]

Looking at the error page as well, gives the following set of combinations:

Firefox 2.0.0.18:

HTML source         status bar     URL bar        error page

%-encoded           SPOOF          %-encoded      %-encoded
UTF8-encoded        Punycode       Punycode       Punycode
entity-encoded      Punycode       Punycode       Punycode

Firefox 3.0.4:

HTML source         status bar     URL bar        error page

%-encoded           SPOOF          SPOOF          %-encoded
UTF8-encoded        Punycode       Punycode       Punycode
entity-encoded      Punycode       Punycode       Punycode

Comment 9

[Neil Harris]

Attachment: Bad percent-encoded URL testcase, in four different encodings

As the previous version, with a Punycoded version in addition to the other three encodings

Comment 10

[Neil Harris]

And, for the sake of completeness, adding a Puncoded version to the source gives the following set of combinations:

Firefox 2.0.0.18:

HTML source         status bar     URL bar        error page

%-encoded           SPOOF          %-encoded      %-encoded
UTF8-encoded        Punycode       Punycode       Punycode
entity-encoded      Punycode       Punycode       Punycode
Punycoded           Punycode       Punycode       Punycode

Firefox 3.0.4:

HTML source         status bar     URL bar        error page

%-encoded           SPOOF          SPOOF          %-encoded
UTF8-encoded        Punycode       Punycode       Punycode
entity-encoded      Punycode       Punycode       Punycode
Punycoded           Punycode       Punycode       Punycode

Comment 11

[Neil Harris]

The relevant logic seems likely to have been introduced in fixing bug 309671: "Support %-escaped hostnames per RFC 3986 (3.2.2)"

It's really difficult to understand how this area of the codebase works, because of its history as an accretion of tweaks, patches and special-case optimizations as its functionality has changed over the years, and the way it is used both for encoding/decoding and for URL obfuscation for securitry reasons; it really needs a complete refactoring.

Comment 12

[Neil Harris]

Confirmed that Firefox 3 actually performs DNS lookup of correctly Punycoded version of spoofed address, so this looks exploitable.

Comment 13

[Neil Harris]

Changed platform to All/All: behaviour seems reproducible on MacOS_X/PPC and Debian/x86.

Comment 14

[Masahiro YAMADA]

According result of trace with debugger, 

nsStandardURL::NormalizeIDN() is called with percent-encoded hostname, and it calles nsIDNService::ConvertToDisplayIDN() without unescaping percent-encode.

And, its stacktrace is

nsStandardURL::NormalizeIDN() line 381
nsStandardURL::BuildNormalizedSpec() line 500
nsStandardURL::SetSpec() line 1079
nsStandardURL::Init() line 2542
NewURI()

I think , BuildNormalizedSpec() or NormalizedIDN() should un-escape percent-encodd.

Comment 15

[Masahiro YAMADA]

(In reply to comment #14)
> 
> I think , BuildNormalizedSpec() or NormalizedIDN() should un-escape
> percent-encodd.

It may be better to decode hostname in ConvertToDisplayIDN.

Comment 16

[Masahiro YAMADA]

Depends on Bug 309671 or DUP?

Comment 17

[Masahiro YAMADA]

Attachment: Workaround for Trunk

This is almostly same as patch for nsStandardURL in bug 309671.

Comment 18

[Masahiro YAMADA]

Attachment: Patch rev.1.0

Added Unit test.

Comment 19

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 368044 [details] [diff] [review]
Patch rev.1.0

I'm not really qualified to review this....

Comment 20

[Christian :Biesinger (don't email me, ping me on IRC)]

(In reply to comment #12)
> Confirmed that Firefox 3 actually performs DNS lookup of correctly Punycoded
> version of spoofed address, so this looks exploitable.

I don't see this happening with Firefox 3.0.7 on Linux. Wireshark does not show a DNS request when clicking on this bug's URL. How did you test this?

Comment 21

[Masahiro YAMADA]

Attachment: new testcase

In this testcase, all links contains different hostname, to make easy to know which link causes to fire DNS query.

Comment 22

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 368044 [details] [diff] [review]
Patch rev.1.0

I assume this is no longer relevant. please re-request if you disagree.

Comment 23

[Patrick McManus [:mcmanus]]

valentin - there is a patch here that biesi timed out.. should we be using it?

Comment 24

[Valentin Gosu [:valentin] (he/him)]

We don't need the bit in nsStandardURL, but having the unit test in the tree would be useful.

Comment 25

[Valentin Gosu [:valentin] (he/him)]

Attachment: Add test to make sure percent encoded host name can't bypass IDN normalization

Also, we can probably clear the security flag, as this is no longer a problem.

Comment 26

[Patrick McManus [:mcmanus]]

Comment on attachment 8716304 [details] [diff] [review]
Add test to make sure percent encoded host name can't bypass IDN normalization

Review of attachment 8716304 [details] [diff] [review]:
-----------------------------------------------------------------

love tests

Comment 27

[Valentin Gosu [:valentin] (he/him)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/54f494f0ecd43f89537f9440629fc7220ba4a3fe
Bug 464591 - Add test to make sure percent encoded host name can't bypass IDN normalization r=mcmanus

Comment 28

[Carsten Book [:Tomcat]]

https://bugzilla.mozilla.org/show_bug.cgi?id=464591