859172 - Firefox now trying to mistakenly decode non-UTF8 links as UTF-8 in UTF-8 webpages

Status: VERIFIED FIXED

(Core :: Internationalization, defect)

Description

[fireattack [:fireattack]]

I just made two web pages to reproduce this "change" or "problem" starting from fx 20.

http://plaza.ufl.edu/bpeng/ansi.html
http://plaza.ufl.edu/bpeng/utf8.html

They're ANSI and UTF-8 encoding htmls, respectively. The contents are same: just two links, the ANSI and UTF-8 URL encoding of Chinese word "闪之轨迹" (%C9%C1%D6%AE%B9%EC%BC%A3 and %e9%97%aa%e4%b9%8b%e8%bd%a8%e8%bf%b9).

In my Chinese win7 OS, in that ANSI html file, when I mouse over both links, it will show “http://www.example.com/闪之轨迹/” correctly at left bottom corner.

However in that UTF-8 html file, The utf-8 link is obviously will work correctly. But the ANSI link, will show as "http://www.example.com/��֮�켣/". Apparently it's the result that Firefox trying decoding it as UTF-8.

In Firefox 19 and earlier version, it will intelligently not bother to decode the link with ANSI strings in any UTF-8 htmls (a.k.a. it will just display it as http://www.example.com/%C9%C1%D6%AE%B9%EC%BC%A3/ if you mouse over). I dunno why it distinguish but it does.

Comment 1

[fireattack [:fireattack]]

Oh I should mention that it's just about the display problem in link tooltip (and in history or location bar search). I can still visit these links w/o trouble.

Comment 2

[Loic]

With FF20 on French Win 7, in both html files, I saw in the tooltip:
http://www.example.com/闪之轨迹/ (UTF-8 Encoding)
http://www.example.com/��֮�켣/ (ANSI (GB) Encoding)

With FF19:
http://www.example.com/闪之轨迹/
http://www.example.com/%C9%C1%D6%AE%B9%EC%BC%A3/

Regression range:
m-c
good=2012-12-11
bad=2012-12-12
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4dfe323a663d&tochange=634180132e68

Suspected bug:
Bug 638379 - make nsIUnicodeDecoder::SetInputErrorBehavior reliable and stop using nsIUnicodeDecoder::Reset for error handling

Comment 3

[Alice0775 White]

Regression window(m-i)
Good:
http://hg.mozilla.org/integration/mozilla-inbound/rev/21d667b49f3a
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20121210 Firefox/20.0 ID:20121210060651
Bad:
http://hg.mozilla.org/integration/mozilla-inbound/rev/a4c6ce2b95a1
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20121210 Firefox/20.0 ID:20121210061551
Pushlog:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=21d667b49f3a&tochange=a4c6ce2b95a1

In local build:
Last Good: 21d667b49f3a
First Bad: 78b0678417f9

Regressed by:
	78b0678417f9	Masatoshi Kimura — Bug 638379 - Part 1: Implement kOnError_Recover to the UTF-8 decoder. r=smontagu

Comment 4

[Ada [:adalucinet]]

This issue is also reproducible on Mac OSX 10.8.3 and Ubuntu 12.10 (32 bit). Changing platform to all.

Comment 5

[Masatoshi Kimura [:emk]]

Attachment: Call SetInputErrorBehavior from convertURItoUnicode

convertURItoUnicode assumes nsIUnicodeDecoder::Convert will fail which will no longer hold unless SetInputErrorBehavior is called.

Comment 6

[bhavana bajaj [:bajaj]]

tracking and passing this on to emk as this is a recent regression from Bug 638379.Please reassign if needed.

Comment 7

[bhavana bajaj [:bajaj]]

: emk, I have pinged Simon to help with the review request here so this can land timely on m-c, aurora and get the required verification.Hopefully we will have it soon :)

Uplift on beta needs to happen by Monday & depends on the risk evaluation of the patch here.

Comment 8

[Masatoshi Kimura [:emk]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ffa0f671afd9

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ffa0f671afd9

Comment 10

[Masatoshi Kimura [:emk]]

Comment on attachment 734879 [details] [diff] [review]
Call SetInputErrorBehavior from convertURItoUnicode

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 638379
User impact if declined: URL will be garbled (display only).
Testing completed (on m-c, etc.): on m-c
Risk to taking this patch (and alternatives if risky): Very low.
String or IDL/UUID changes made by this patch: none

Comment 11

[bhavana bajaj [:bajaj]]

(In reply to Benjamin Peng from comment #1)
> Oh I should mention that it's just about the display problem in link tooltip
> (and in history or location bar search). I can still visit these links w/o
> trouble.

Thanks for your report. The issue is resolved should be resolved in our latest nightly and will be uplifted all the way to Fx21 in Fx 21 beta 4. 

Can you verify one of our latest nightly or aurora (once the patch lands) that the issue is resolved for you ? Thanks !

Comment 12

[Masatoshi Kimura [:emk]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/f4843d478646
https://hg.mozilla.org/releases/mozilla-beta/rev/018d1dfbcf2e

Comment 13

[Ada [:adalucinet]]

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0
Mozilla/5.0 (X11; Linux i686; rv:21.0) Gecko/20100101 Firefox/21.0

Verified fixed on Firefox 21 beta 4 (Build ID: 20130423212553), latest Nightly (Build ID: 20130424030917) and latest Aurora (Build ID: 20130424004015). When I mouse over the both links from comment 0, the result is as expected:
http://www.example.com/闪之轨迹/
http://www.example.com/%C9%C1%D6%AE%B9%EC%BC%A3/

Comment 14

[Ada [:adalucinet]]

Mozilla/5.0 (X11; Linux i686; rv:22.0) Gecko/20100101 Firefox/22.0
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0

Verified as fixed on FF 22 beta 1 (Build Id: 20130514181517) en-Us and fr builds.

Comment 15

[Cornel Ionce [:noni] [Hubs QA]]

Verified as fixed on FF 23 beta 10 (build ID: 20130729175331) using the webpages from comment 0. 

User Agents:
Mozilla/5.0 (X11; Linux i686; rv:23.0) Gecko/20100101 Firefox/23.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:23.0) Gecko/20100101 Firefox/23.0
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0