Closed Bug 859172 Opened 7 years ago Closed 7 years ago

Firefox now trying to mistakenly decode non-UTF8 links as UTF-8 in UTF-8 webpages

Categories

(Core :: Internationalization, defect)

20 Branch
defect
Not set

Tracking

()

VERIFIED FIXED
mozilla23
Tracking Status
firefox20 --- affected
firefox21 + verified
firefox22 + verified
firefox23 + verified

People

(Reporter: human.peng, Assigned: emk)

References

Details

(Keywords: regression)

Attachments

(1 file)

I just made two web pages to reproduce this "change" or "problem" starting from fx 20.

http://plaza.ufl.edu/bpeng/ansi.html
http://plaza.ufl.edu/bpeng/utf8.html

They're ANSI and UTF-8 encoding htmls, respectively. The contents are same: just two links, the ANSI and UTF-8 URL encoding of Chinese word "闪之轨迹" (%C9%C1%D6%AE%B9%EC%BC%A3 and %e9%97%aa%e4%b9%8b%e8%bd%a8%e8%bf%b9).

In my Chinese win7 OS, in that ANSI html file, when I mouse over both links, it will show “http://www.example.com/闪之轨迹/” correctly at left bottom corner.

However in that UTF-8 html file, The utf-8 link is obviously will work correctly. But the ANSI link, will show as "http://www.example.com/��֮�켣/". Apparently it's the result that Firefox trying decoding it as UTF-8.

In Firefox 19 and earlier version, it will intelligently not bother to decode the link with ANSI strings in any UTF-8 htmls (a.k.a. it will just display it as http://www.example.com/%C9%C1%D6%AE%B9%EC%BC%A3/ if you mouse over). I dunno why it distinguish but it does.
Oh I should mention that it's just about the display problem in link tooltip (and in history or location bar search). I can still visit these links w/o trouble.
With FF20 on French Win 7, in both html files, I saw in the tooltip:
http://www.example.com/闪之轨迹/ (UTF-8 Encoding)
http://www.example.com/��֮�켣/ (ANSI (GB) Encoding)

With FF19:
http://www.example.com/闪之轨迹/
http://www.example.com/%C9%C1%D6%AE%B9%EC%BC%A3/

Regression range:
m-c
good=2012-12-11
bad=2012-12-12
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4dfe323a663d&tochange=634180132e68

Suspected bug:
Bug 638379 - make nsIUnicodeDecoder::SetInputErrorBehavior reliable and stop using nsIUnicodeDecoder::Reset for error handling
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: regression
Regression window(m-i)
Good:
http://hg.mozilla.org/integration/mozilla-inbound/rev/21d667b49f3a
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20121210 Firefox/20.0 ID:20121210060651
Bad:
http://hg.mozilla.org/integration/mozilla-inbound/rev/a4c6ce2b95a1
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20121210 Firefox/20.0 ID:20121210061551
Pushlog:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=21d667b49f3a&tochange=a4c6ce2b95a1

In local build:
Last Good: 21d667b49f3a
First Bad: 78b0678417f9

Regressed by:
	78b0678417f9	Masatoshi Kimura — Bug 638379 - Part 1: Implement kOnError_Recover to the UTF-8 decoder. r=smontagu
This issue is also reproducible on Mac OSX 10.8.3 and Ubuntu 12.10 (32 bit). Changing platform to all.
OS: Windows 7 → All
Hardware: x86_64 → All
Component: Location Bar → Internationalization
Product: Firefox → Core
convertURItoUnicode assumes nsIUnicodeDecoder::Convert will fail which will no longer hold unless SetInputErrorBehavior is called.
Assignee: nobody → VYV03354
Status: NEW → ASSIGNED
Attachment #734879 - Flags: review?(smontagu)
tracking and passing this on to emk as this is a recent regression from Bug 638379.Please reassign if needed.
Status: ASSIGNED → NEW
: emk, I have pinged Simon to help with the review request here so this can land timely on m-c, aurora and get the required verification.Hopefully we will have it soon :)

Uplift on beta needs to happen by Monday & depends on the risk evaluation of the patch here.
Attachment #734879 - Flags: review?(smontagu) → review+
https://hg.mozilla.org/mozilla-central/rev/ffa0f671afd9
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
Comment on attachment 734879 [details] [diff] [review]
Call SetInputErrorBehavior from convertURItoUnicode

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 638379
User impact if declined: URL will be garbled (display only).
Testing completed (on m-c, etc.): on m-c
Risk to taking this patch (and alternatives if risky): Very low.
String or IDL/UUID changes made by this patch: none
Attachment #734879 - Flags: approval-mozilla-beta?
Attachment #734879 - Flags: approval-mozilla-aurora?
Attachment #734879 - Flags: approval-mozilla-beta?
Attachment #734879 - Flags: approval-mozilla-beta+
Attachment #734879 - Flags: approval-mozilla-aurora?
Attachment #734879 - Flags: approval-mozilla-aurora+
(In reply to Benjamin Peng from comment #1)
> Oh I should mention that it's just about the display problem in link tooltip
> (and in history or location bar search). I can still visit these links w/o
> trouble.

Thanks for your report. The issue is resolved should be resolved in our latest nightly and will be uplifted all the way to Fx21 in Fx 21 beta 4. 

Can you verify one of our latest nightly or aurora (once the patch lands) that the issue is resolved for you ? Thanks !
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0
Mozilla/5.0 (X11; Linux i686; rv:21.0) Gecko/20100101 Firefox/21.0

Verified fixed on Firefox 21 beta 4 (Build ID: 20130423212553), latest Nightly (Build ID: 20130424030917) and latest Aurora (Build ID: 20130424004015). When I mouse over the both links from comment 0, the result is as expected:
http://www.example.com/闪之轨迹/
http://www.example.com/%C9%C1%D6%AE%B9%EC%BC%A3/
Mozilla/5.0 (X11; Linux i686; rv:22.0) Gecko/20100101 Firefox/22.0
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:22.0) Gecko/20100101 Firefox/22.0

Verified as fixed on FF 22 beta 1 (Build Id: 20130514181517) en-Us and fr builds.
Verified as fixed on FF 23 beta 10 (build ID: 20130729175331) using the webpages from comment 0. 

User Agents:
Mozilla/5.0 (X11; Linux i686; rv:23.0) Gecko/20100101 Firefox/23.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:23.0) Gecko/20100101 Firefox/23.0
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0
Status: RESOLVED → VERIFIED
Keywords: verifyme
You need to log in before you can comment on or make changes to this bug.