Closed Bug 315381 Opened 15 years ago Closed 14 years ago

accents incorrectly encoded - decoded

Categories

(Core :: Internationalization, defect)

x86
Windows XP
defect
Not set

Tracking

()

VERIFIED FIXED

People

(Reporter: fidelardo, Assigned: smontagu)

References

()

Details

(Keywords: fixed1.8.1, verified1.8.0.5)

Attachments

(4 files)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051025 Firefox/1.5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051025 Firefox/1.5

When writting in spanish language in Yahoo mail, and you "save as draft" or "attach files" the accents (or special characters like ñ) that were previously entered (and shown correctly) transform into ? characters. It seem to be a encoding - decoding problem. Keep getting the same problem in other sites too.
I tried to change encoding with no results. This problem is not present in the 1.0.7 version of Firefox, only in 1.5 (beta 2, and RC1). Using Western ISO-8859-1 enconding.

Reproducible: Always

Steps to Reproduce:
1.login to yahoo mail
2.compose mail message with accents (spanish) á é í ó ú  and special characters like ñ
3.Press "attach file" and then "cancel" to return to the message. 
 

Actual Results:  
Then the accents and the special characters are transformed into ? characters.

Expected Results:  
The accents and special characters may be shown correctly.
Version: unspecified → 1.5 Branch
I'm experiencing something similar in Yahoo Mail as well.  The behaviour described   by Fiddi and also this:

- Read an e-mail with accented characters
- Hit Reply
- The reply window contains "?" characters instead of the accents

I believe this started in Firefox 1.5 RC2 but I'm not sure.  It's definitely there in RC3.  This was working fine in 1.0.7.  I also tested this in IE and Opera since Yahoo had recently made changes to Yahoo Mail, but they both work fine.
UPDATE:

I installed the final version of 1.5 today, but instead of using the auto-updater, I downloaded the installer and did a full reinstall (but did not remove RC3 before that.)  Yahoo Mail now works correctly, and that's even though RC3 and Final are apparently identical(!)  Could this have something to do with the auto-update (from RC2 to RC3)?  A bit far-fetched, I admit.
*** Bug 319160 has been marked as a duplicate of this bug. ***
I just installed the final release of Firefox 1.5 (fresh install), unfortunatelly I keep getting the same errors. That tends to be an annoying bug. Weird it was not happening with Firefox 1.0.x.
(In reply to comment #4)
> I just installed the final release of Firefox 1.5 (fresh install),
> unfortunatelly I keep getting the same errors. That tends to be an annoying
> bug. Weird it was not happening with Firefox 1.0.x.
> 

The problem *I* was having is fixed, but I tried what you described (Attach files) and get the same problems with the accents.
(In reply to comment #0)

I have the same exact problem with French accented characters in Yahoo mail, i did NOT have this problem in Firefox 1.0.7 or 1.0.4 ...
For me, this bug should be taken very seriously, it still has not been fixed 1.5.01, while it makes firefor unusable for me....

Very simple example:
http://www.rbcbanqueroyale.com/

The accented characters are completely messed up... Now i have to use Internet Explorer for my web sites!!! ;(

This problem is really anoying and disgusting, many people comments they have been experiencing the same problem. Today I installed the nigthy build of "Deer Park alpha 2" (aka firefox 1.6) which in fact has the same problem, incredible?
If it continues... i will have to dispose firefox :S is completelly unusable for me... and for all who has to deal with accents (specially Latin languages - Italian, Portuguese, Spanish, French... and so on).

Please take further action.

I totally agree! I have to use IE to write my emails!
I used to love firefox, but this is annyoing for me too!!!! Looks like the Firefox people think everybody uses English as their primary language, well it's not my case and I need the software to work correctly using accent marks. So what do you think? Do you consider people speaking any other languages than English important as firefox users?
Component: General → Internationalization
Product: Firefox → Core
Version: 1.5.0.x Branch → Trunk
Assignee: nobody → smontagu
QA Contact: general → amyy
*** Bug 331859 has been marked as a duplicate of this bug. ***
I am very annoyed too with this long-time bug since version 1.5 (Firefox 1.0.7 is does not have the bug).

Can someone take action as soon as possible on it, before the next version of Firefox is out?

I am a long-time fan of Firefox but now after several months living with this bug on a day-to-day basis, I'm considering switching back to IE.  This bug is really a pain in the ass.  I speak and write french as my primary language.
Does someone is actively working on this important bug (but unfortunately not "confirmed" yet) ?

Thanks in advance.
I can't reproduce this bug. I can achieve the effect in attachment 217581 [details] by manually changing encoding to UTF-8, but not in normal use.
This is my settings in the "Languages" dialog to be able to reproduce the bug.
What is your default character encoding (from the Advanced fonts preference dialog)?
Currently, my default Character encoding (in the Advanced / Fonts preferences) is Western (ISO-8859-1).

I used to try all possible combinaisons in View / Character Encoding with and without "Auto-Detect" and this bug is still happening.
Hi, since we're getting no support at all. I saw on the firefox site that they just released version 1.0.8 which has security fixes found in 1.5. In your main concern is the accents and special characters, switch back to version 1.0.8. You can get the correct copy for your platform here:  http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/1.0.8/

Yes but your solution will not address source of the problem in FF 1.5.

What's changed between FF 1.0.x -> 1.5 at the Codepage / Character Encoding level ?
By now -and being an IT analyst- I feel very dissapointed about the treatment the Firefox community gave to this IMPORTANT BUG.
Do you place any regression tests on the project???

Firefox Development team: Please, take further and responsible actions. Simon Montagu if you can´t manage this problem by yourself please report it in the correct manners. Otherwise you will be loosing a big part of users: those who writes in Latin Languages: Italian, French, Spanish, Portuguese.

i aprecciate your help
thanks a lot
Yes, that's right :  TAKE ACTION.

This is a VERY annoying and IMPORTANT bug!
This bug is way enough to make non-english languages people switch to other browsers and discard Firefox.

Take action, please.
https://bugzilla.mozilla.org/page.cgi?id=etiquette.html

Repeated comments saying "this bug must be fixed" do not do anything to help fix it. I have still been unable to reproduce the bug. Can somebody send me the source of a Yahoo mail page exhibiting the problem? I will also try opening a test Yahoo account and give my location as France or Spain to see if I can reproduce that way.
(In reply to comment #23)
> Can somebody send me the
> source of a Yahoo mail page exhibiting the problem?

Not necessary: I have reproduced the bug in a new Yahoo account.

Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
(In reply to comment #24)
> (In reply to comment #23)
> > Can somebody send me the
> > source of a Yahoo mail page exhibiting the problem?
> 
> Not necessary: I have reproduced the bug in a new Yahoo account.
> 

If you require anymore information, please let me know, i will gladly help. Good luck in the bugfixing!
This is regression from bug 306310, or rather an old bug that was exposed by bug 306310.

I still need more diagnosis as to exactly what the old bug is. It may also be related to the special handling of POST pages in nsHTMLDocument::StartDocumentLoad()
Blocks: 306310
None of the Yahoo mail pages declare any charset.

The Yahoo mail composition page is loaded as the default charset (assume this is ISO-8859-1 for purposes of discussion). The page also contains an iframe with src="about:blank", whose charset is UTF-8. When the iframe is loaded, we go though the code at http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/content/html/document/src/nsHTMLDocument.cpp&rev=3.670&cvsroot=/cvsroot&mark=902-903#897 
However, because of lines 810-815 above, muCV is the contentViewer for the parent document, so the charset of about:blank has now overwritten the charset of the Yahoo page.

Pressing "Attach files" does a POST with the message content in a hidden input, encoded in the actual charset of the composition page (ISO-8859-1) and at lines 877-882 the resulting document is assigned the previous charset from the content viewer (UTF-8).

Pressing "Cancel" does another POST, and this time we try to encode in UTF-8, but the characters > 0x7F from the original message content are not valid UTF-8, so are encoded as replacement characters (the �s in the screenshot).
Attachment #219609 - Flags: superreview?(jst)
Attachment #219609 - Flags: superreview?
Attachment #219609 - Flags: review?(jst)
Attachment #219609 - Flags: review?
Comment on attachment 219609 [details] [diff] [review]
Don't propagate charset of a child frame to the parent's content viewer

Makes sense. r+sr=jst
Attachment #219609 - Flags: superreview?(jst)
Attachment #219609 - Flags: superreview+
Attachment #219609 - Flags: review?(jst)
Attachment #219609 - Flags: review+
Checked in to trunk. Can someone verify that the bug is fixed in the next nightly build?

After this has baked for a while I'll request branch approval.
Flags: blocking1.8.1?
I tried with Minefield 3.0a1 on win32 and the problem is not reoccuring.

I do not know if this is normal, but when I hit "Reply", the HTML email is switched to "Plain text" every time...  This behavior was not happening in FF 1.5.0.2 (HTML format is maintained, but the bug is happening).
That is bug 318665.
Comment on attachment 219609 [details] [diff] [review]
Don't propagate charset of a child frame to the parent's content viewer

I think this has baked long enough on the trunk. Do we want it for 1.8.0.x as well?
Attachment #219609 - Flags: approval-branch-1.8.1?(jst)
Flags: blocking1.8.0.5?
I know that this is a bit off-topic from the core of bug 315381, but I sincerely wish to thank you very much Simon for fixing this bug.  It is really appreciated, great work!

Just for my personal culture, do you know if this fix could be included in the next version of Firefox, the 1.5.0.4 ?
Attachment #219609 - Flags: approval-branch-1.8.1?(jst) → approval-branch-1.8.1+
Keywords: fixed1.8.1
Comment on attachment 219609 [details] [diff] [review]
Don't propagate charset of a child frame to the parent's content viewer


Checkin: {
2006-05-16 03:45    smontagu%smontagu.org mozilla/content/html/document/src/nsHTMLDocument.cpp     3.615.2.19 MOZILLA_1_8_BRANCH
}
No longer blocks: 315957
*** Bug 323295 has been marked as a duplicate of this bug. ***
Don't know if it is related, but when you save a web site that contains any of these special characters as a text file (txt), the characters are improperly saved.
To test this:
go to 
http://es.wikisource.org/w/index.php?title=Miguel_Strogoff:_Primera_parte:_Cap%C3%ADtulo_I&printable=yes
right click, select save page as...
select Text File in the Save as File drop down box
Open the resulting file using any text editor BUT Notepad
the special characters will be represented by two weird characters( i.e 
ó becomes ó , ñ --> ñ , etc
That is not related, and not a bug. The example you give is encoded in UTF-8, so when it is saved as a text file, you need to open it in an editor that supports UTF-8 (which is why it works in Notepad, for example).
Flags: blocking1.8.0.5? → blocking1.8.0.5+
Comment on attachment 219609 [details] [diff] [review]
Don't propagate charset of a child frame to the parent's content viewer

approved for 1.8.0 branch, a=dveditz for drivers
Attachment #219609 - Flags: approval1.8.0.5?
Attachment #219609 - Flags: approval1.8.0.5?
Checked in to MOZILLA_1_8_0_BRANCH.
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Flags: blocking1.8.1?
Flags: blocking1.8.0.5+
Keywords: fixed1.8.0.5
Resolution: --- → FIXED
This happens in Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060516 SeaMonkey/1.0.2
Comment 41 doesn't seem to be related to this bug. It's interesting that "España" is displayed correctly in the headlines. Can you provide sample URLs?
Verified on 1.5.0.5 nightly builds using:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060619 Firefox/1.5.0.5
Status: RESOLVED → VERIFIED
Verified by using the test case provided, composing an email with accents and special characters, then "attach file" and canceling. Also tried replying to an email with these characters. I also checked the fix in the Mac version which also had the same problem.
Simon, do you think that it's related to this bug?

http://linux.startcom.org/?lang=fr&app=15
That site is just misconfigured: it declares the encoding as UTF-8 in both HTTP headers and <META http-equiv="content-type">, but is actually in ISO-8859-1 or similar.
*** Bug 344741 has been marked as a duplicate of this bug. ***
You need to log in before you can comment on or make changes to this bug.