1142083 - IDN Unicode domain redirect is broken in Firefox 36/37/38

Status: RESOLVED FIXED

(Core :: Networking: DNS, defect, P1)

Description

[KestutisIT]

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0
Build ID: 20150311004633

Steps to reproduce:

1. Enter http://zymiausifotografai.lt
2. Believe that should be redirected to http://www.žymiausifotografai.lt/


Actual results:

You get 404 (problem loading page) 'Server not found' error:
Firefox can't find the server at www.%c5%beymiausifotografai.lt.



Expected results:

You should be redirected to http://www.žymiausifotografai.lt/

Appears url do not converts to IDN name:
http://www.xn--ymiausifotografai-wzd.lt 
https://iwantmyname.com/domain-tools/idns/idn-punycode-converter?utf8=http%3A%2F%2Fwww.%C5%BEymiausifotografai.lt+&search=convert

It works well in Internet Explorer 11 and Google Chrome newest.

Comment 1

[KestutisIT]

Attachment: bug-idn.jpg

Bug report.

Comment 2

[Mike Mueller]

I reproduced this with a local build and was able to fix it by commenting out a NS_EscapeURL call from nsHttpChannel.cpp:

nsresult                                                                                  
nsHttpChannel::AsyncProcessRedirection(uint32_t redirectType)
{   
    ...

    // make sure non-ASCII characters in the location header are escaped.                 
    nsAutoCString locationBuf;
    if (NS_EscapeURL(location, -1, esc_OnlyNonASCII, locationBuf))
        location = locationBuf.get();

Obviously the issue is that the host part of a URI should not be escaped using URL-encoding. That is causing the hostname to be incorrect (having %c5%be, etc.) in the subsequent request.

Does anyone know if this escaping is actually needed? In NewURI(), a nsStandardURL is constructed from the location; does that class handle Unicode URIs already?

Comment 3

[Mike Hoye [:mhoye]]

Good catch, both of you. Let me needinfo jduell, and see if he can sort this out.

Comment 4

[Jason Duell]

This smells like something Valentin ought to know how to fix.

Comment 5

[Jason Duell]

Valentin, any thoughts here?  If you don't know we can find someone else.

Comment 6

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

(In reply to Jason Duell [:jduell] (needinfo? me) from comment #5)
> Valentin, any thoughts here?  If you don't know we can find someone else.

Yes, I've been looking into this.
Just commenting NS_EscapeURL triggers an assertion in NSS, so that's probably not good.
This will be fixed by bug 1023468 in which we start doing percent decoding for hostnames (a very requested feature).

Comment 7

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

Attachment: Add test for IDN Unicode domain redirect

This is a patch to make sure we don't regress this feature. Once bug 1023468 lands, IDN domain redirect should work.

Comment 8

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

Attachment: IDN Unicode domain redirect is broken

It might still be a while until we manage to land hostname percent decoding, so this a simple fix just for this one bug

Comment 9

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

Attachment: Add test for IDN Unicode domain redirect

This test still has a slight problem - my guess is that there is a problem in how httpd.js handles non-ascii headers.

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6c782e3a7078

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6c782e3a7078

Comment 12

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

Attachment: Add test for IDN Unicode domain redirect

Comment 13

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

Awaiting a green inbound in order to land the test.

Comment 14

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4eac2966dab7

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4eac2966dab7

Comment 17

[Al Billings [:abillings - ex-MoCo]]

This is now unfixed on beta and aurora as it was backed out elsewhere.

Comment 18

[Kohei Yoshino [:kohei]]

Posted the site compatibility doc for reference: https://www.fxsitecompat.com/en-US/docs/2015/idn-urls-are-not-redirected-properly/

[Tracking Requested - why for this release]: See Comment 17.

Comment 19

[Sylvestre Ledru [:Sylvestre]]

Valentin, tomorrow, we are going to build beta 9. I guess this is going to be hard to get a fix, correct?

Release Note Request (optional, but appreciated)
[Why is this notable]: regression
[Suggested wording]: URLs containing a Unicode-format Internationalized Domain Name (IDN) are not redirected properly, leading to a Server Not Found error.
[Links (documentation, blog post, etc)]:

Comment 20

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

(In reply to Sylvestre Ledru [:sylvestre] from comment #19)
> Valentin, tomorrow, we are going to build beta 9. I guess this is going to
> be hard to get a fix, correct?

The fix will likely be a bit complex, and definitely not suitable for beta.
This was also talked about in bug 439616, but it hadn't reached a conclusion.
I'll try to find a smarter way to fix this.

Comment 21

[Sylvestre Ledru [:Sylvestre]]

OK, then, firefox 42 is ship with this bug. Please try to provide a fix for fx 43.
Added to the list of know issues in 42.

Comment 22

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0757ced63723

Comment 23

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

https://hg.mozilla.org/integration/mozilla-inbound/rev/18b0d63d9f06d849a8358e5acff33f7fc70250f8
Bug 1142083 - Add test for IDN Unicode domain redirect r=mcmanus

Comment 24

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/18b0d63d9f06

Comment 25

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Valentin, does this need to be uplifted to Beta43 and Aurora44?

Comment 26

[Valentin Gosu [:valentin] (he/him) {{ FYI: OOO 14 June - 4 August }}]

The patch contains only the test cases. The actual fix is in bug 412457.
Since this is a fairly large change, and the IDN redirect has been an issue for multiple years, I would rather let it ride the trains.

Comment 27

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

(In reply to Valentin Gosu [:valentin] from comment #26)
> The patch contains only the test cases. The actual fix is in bug 412457.
> Since this is a fairly large change, and the IDN redirect has been an issue
> for multiple years, I would rather let it ride the trains.

Thanks Valentin. In that case, we will wontfix this for FF44.

Comment 29

[Liz Henry (:lizzard) (relman/hg->git project)]

Wontfix for 43. Sounds like we should also consider adding a release note for the fix for 45.

Comment 30

[Sylvestre Ledru [:Sylvestre]]

Yes, added to the release notes with "URLs containing a Unicode-format Internationalized Domain Name (IDN) are now properly redirected" as wording for 45 (marked as fixed)