Closed Bug 190951 Opened 17 years ago Closed 17 years ago

Japanese Description of Helper App. will be destroyed after restart if default encoding pref is set appropriately


(Core :: XML, defect, critical)

Not set





(Reporter: u69748, Assigned: dbaron)



(Keywords: dataloss, Whiteboard: [patch])


(5 files)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b) Gecko/20030126
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b) Gecko/20030126

Japanese Description of Helper App. will be destroyed after restart.
File opening dialog has also same problem.

Reproducible: Always

Steps to Reproduce:
1. Download a zip file
2. Restart Mozilla
3. See Description of zip file in Preferences->Nvigator->
   Helper Applications. 

Actual Results:  
Japanese description is destroyed.

Expected Results:  
show correct description.
Assignee: asa → law
Component: Browser-General → File Handling
QA Contact: asa → petersen

*** This bug has been marked as a duplicate of 189106 ***
Closed: 17 years ago
Resolution: --- → DUPLICATE
Bug 189106 has been fixed.
This is not fixed.

Resolution: DUPLICATE → ---
This problem is reproduce still.(2003020408-trunk/WinXP)

This problem is very serious.
Because, this problem destroy "mimeTypes.rdf".

When I add a MIME type with Japanease Description and Close Mozilla,
Mozilla saved to mimeTypes.rdf correctly.
But,When I restart Mozilla, Mozilla can't read the file correctry.
And I re-saved the file, Japanease Description in the file was destroyed.
Flags: blocking1.3b?
OK.  Could you please do the following for me?

1)  Start Mozilla with a clean profile (use mozilla -ProfileManager)
2)  Go to the helper app preferences
3)  Click the "New" button; add one entry with a description in Japanese.
4)  Click that entry in the list, verify that the description shows correctly.
5)  Quit Mozilla.
6)  Attach the resulting mimeTypes.rdf to this bug
7)  Start Mozilla with the same profile
8)  Open the helper app preferences
9)  Select the entry you added.  Do _not_ edit it; just look at the description
    under the listbox.
10) Verify that the description is now incorrect
11) Quit Mozilla
12) Attach the mimeTypes.rdf you see at that point to this bug if the desription
    was incorrect in step 10.  Otherwise, just comment that the description was
One other thing.  Are you able to test this on things other than Windows?  I can
only test on Linux, and I have been unable to reproduce it, but it could be
Windows-only in some bizarre way (not sure how; all the mimeTypes.rdf code is
I can't reproduce this problem with New profile.
But, I found other relation.
When I changed to Japanese(EUC-JP) or Japanese(ISO-2022-JP) at [Navigator] -> 
[Languages] -> Character Coding, I could reproduce this problem.
However, When Japanese(Shift_JIS) was setted, this problem can't reproduce.

#Shift_JIS is Japanese default character code on Windows and Mac.
#EUC-JP is Japanese default character code on UNIX(not include HP-UX).
Um... _none_ of the code that touches mimeTypes.rdf should care what the default
character coding is....

So if you create a new profile and then set the default character coding to
Japanese(EUC-JP) or Japanese(ISO-2022-JP) then the problem starts happening in
that new profile?
And just to clarify... this is a description that you enter yourself, not a
description we pick up from the operating system, right?
1. Created new profile.
2. Opened preferences dialog, and added MIME type _myself_.
3. Quited Mozilla.
4. Restarted Mozilla with same profile, and changed character coding to Shift_JIS.
5. Quited Mozilla.
6. Restarted Mozilla with same profile, and changed character coding to EUC-JP.
7. Quited Mozilla.
8. Restarted Mozilla with same profile.

I can show the destroyed Japanese description in step 8.
And after step 8, I added another profile and quited mozilla,
mimeTypes.rdf was destroyed.
I retry the same test with picked up from the operating system.
I can reproduce similarly.

Boris Zbarsky:
There is no relation System/ourself.
I see...

Could you attach the mimeTypes.rdf after each of those steps?  Did the
datasource get broken even when you did _not_ look at the helper app pref panel?
Attachment #113631 - Attachment mime type: text/rdf → text/plain
I show the file always.
I can see the destroyed Japanese description in Prefs panel at first.
And this time, Mozilla saved to the file with the destroyed Japanese description.
And Mozilla restart with the broken file,
I can't see correctly the Japanese Description.

# I re-tested with daily use prfile(Created with 20030112 build),
# When the character coding is Shift_JIS,
# I can see the Japanese Description destroyed...(the file is _not_ destroyed)

# Excuse me. I have to go to work, because now is the morning in Japan.
OK, I've gotten this to break on Linux too.  Exactly as you say: EUC-JP and
ISO-2022-JP get munged, Shift_JIS does not.

Thank you!  Trying to debug this now.  :)

ccing some people who may know why this setting would make _any_ difference in
reading/writing an RDF datasource...
Assignee: law → bzbarsky
Severity: normal → major
Ever confirmed: true
OS: Windows XP → All
Priority: -- → P1
Hardware: PC → All
Summary: Japanese Description of Helper App. will be destroyed after restart → Japanese Description of Helper App. will be destroyed after restart if default encoding pref is set appropriately
Target Milestone: --- → mozilla1.3beta
More weirdness.  If the profile manager comes up, the problem does _not_ show
up.  It only shows up if I start mozilla with 'mozill -P whatever' (or possibly
if you only have one profile).

It's pretty low-level; I'm seeing the wrong strings in the RDF content sink....
OK, more information.

1)  When there is no profile manager there is a very early attempt to use the
    MIME service; this attempt loads the data source and this is when things
2)  When I get a profile manager window, there is no such early attempt
3)  I traced back the bogus string (I could tell it's bogus because it claims to
    be 27 PRUnichars long instead of 9) as far back as Driver_HandleStartElement
    in nsExpatDriver.cpp.  At this point the data is _already_ wrong.
4)  smontagu suggested that based on the screenshots it looks like the data is
    being converted UCS2 -> UTF8 -> byte-pad -> pretend it's UCS2 -> UTF8 or
    something like that.  That would certainl lead to the ballooning from 9 to
    27 chars.

I tried to trace the expat code, and I'm way out of my depth there and my
debugger does not deal with starting without the profile manager (yay gdb).  

dbaron suggested that perhaps some function is being used that is locale-aware,
but I didn't think we changed the locale based on the default encoding setting....

Over to heikki, who may have some idea of what expat is doing.  :(

Raising severity to critical, since this is dataloss and since we sorta rely on
expat working correctly all over the place...
Assignee: bzbarsky → heikki
Severity: major → critical
Component: File Handling → XML
Keywords: dataloss
Priority: P1 → --
QA Contact: petersen → ashishbhatt
Target Milestone: mozilla1.3beta → ---
Blocks: 191183
Attachment #113631 - Attachment filename: コピー 〜 mimeTypes.rdf → �R�s�[ �` mimeTypes.rdf
Attachment #113631 - Attachment mime type: text/plain → text/plain;charset=UTF-8
Attachment #113632 - Attachment mime type: text/plain → text/plain;charset=UTF-8
It looks to me like the data are already wrong in nsScanner::Append, and the
mUnicodeDecoder there is an ISO-8859-1 decoder (it's certainly an
nsOneByteDecoderSupport, and the table looks like Windows-1252 after a 10-second
The problem is that calias->GetPreferred(aCharset, charsetName); in
nsScanner::SetDocumentCharset is failing.
Here's the problem (note how the top is the same as 19 frames down):

#0  nsStringBundle::GetStringFromName(unsigned short const*, unsigned short**)
    (this=0x8501bb0, aName=0xbfffad60, aResult=0x8508bc0)
    at /builds/trunk/mozilla/intl/strres/src/nsStringBundle.cpp:254
#1  0x427312ce in nsURLProperties::Get(nsAString const&, nsAString&) (
    this=0x8505a38, aKey=@0xbfffad50, oValue=@0xbfffae50)
    at /builds/trunk/mozilla/intl/uconv/src/nsURLProperties.cpp:78
#2  0x42730ad2 in nsCharsetAlias2::GetPreferred(nsAString const&, nsAString&) (
    this=0x829a408, aAlias=@0x850b088, oResult=@0xbfffae50)
    at /builds/trunk/mozilla/intl/uconv/src/nsCharsetAliasImp.cpp:107
#3  0x41f9a3cf in nsScanner::SetDocumentCharset(nsAString const&, int) (
    this=0x850a7d8, aCharset=@0x850b088, aSource=14)
    at /builds/trunk/mozilla/htmlparser/src/nsScanner.cpp:211
#4  0x41f99fad in nsScanner (this=0x850a7d8, aFilename=@0xbfffafa0,
    aCreateStream=0, aCharset=@0x850b088, aSource=14)
    at /builds/trunk/mozilla/htmlparser/src/nsScanner.cpp:157
#5  0x41f90e7a in nsParser::Parse(nsIURI*, nsIRequestObserver*, int, void*,
nsDTDMode) (this=0x850afd8, aURL=0x8509a60, aListener=0x0, aVerifyEnabled=0,
    aKey=0x0, aMode=eDTDMode_autodetect)
    at /builds/trunk/mozilla/htmlparser/src/nsParser.cpp:1497
#6  0x41bd1efe in nsRDFXMLParser::ParseAsync(nsIRDFDataSource*, nsIURI*,
nsIStreamListener**) (this=0x84feb60, aSink=0x85085d0, aBaseURI=0x8509a60,
    at /builds/trunk/mozilla/rdf/base/src/nsRDFXMLParser.cpp:86
#7  0x41bcd6d0 in RDFXMLDataSourceImpl::Refresh(int) (this=0x85085d0,
    at /builds/trunk/mozilla/rdf/base/src/nsRDFXMLDataSource.cpp:938
#8  0x41bc8e6e in RDFServiceImpl::GetDataSource(char const*, int,
nsIRDFDataSource**) (this=0x82145c8,
"file:///home/dbaron/.mozilla/190951/nqya4zyn.slt/mimeTypes.rdf", aBlock=1,
    at /builds/trunk/mozilla/rdf/base/src/nsRDFService.cpp:1561
#9  0x41bc8715 in RDFServiceImpl::GetDataSourceBlocking(char const*,
nsIRDFDataSource**) (this=0x82145c8,
    at /builds/trunk/mozilla/rdf/base/src/nsRDFService.cpp:1490
#10 0x41ca48e1 in nsExternalHelperAppService::InitDataSource() (this=0x829e070)
    at /builds/trunk/mozilla/uriloader/exthandler/nsExternalHelperAppService.cpp:231
#11 0x41ca7097 in nsExternalHelperAppService::GetMIMEInfoForExtensionFromDS(char
const*, nsIMIMEInfo**) (this=0x829e070,
    aFileExtension=0xbfffb990 "properties", aMIMEInfo=0xbfffb930)
    at /builds/trunk/mozilla/uriloader/exthandler/nsExternalHelperAppService.cpp:685
#12 0x41cac7f4 in nsExternalHelperAppService::GetFromExtension(char const*,
nsIMIMEInfo**) (this=0x829e070, aFileExt=0xbfffb990 "properties",
#13 0x41cacdd2 in nsExternalHelperAppService::GetTypeFromExtension(char const*,
char**) (this=0x829e070, aFileExt=0xbfffb990 "properties",
#14 0x41cad62a in nsExternalHelperAppService::GetTypeFromFile(nsIFile*, char**)
    (this=0x829e070, aFile=0x8507838, aContentType=0x85018a0)
#15 0x40f59407 in nsFileChannel::EnsureStream() (this=0x8506f68)
    at /builds/trunk/mozilla/netwerk/protocol/file/src/nsFileChannel.cpp:127
#16 0x40f59f5c in nsFileChannel::Open(nsIInputStream**) (this=0x8506f68,
    at /builds/trunk/mozilla/netwerk/protocol/file/src/nsFileChannel.cpp:349
#17 0x41c3912a in NS_OpenURI (result=0xbfffbcb0, uri=0x8506348, ioService=0x0,
    loadGroup=0x0, notificationCallbacks=0x0, loadAttributes=0)
    at ../../../dist/include/necko/nsNetUtil.h:205
#18 0x41c33bea in nsStringBundle::LoadProperties() (this=0x8501bb0)
    at /builds/trunk/mozilla/intl/strres/src/nsStringBundle.cpp:124
#19 0x41c3448a in nsStringBundle::GetStringFromName(unsigned short const*, unsig
ned short**) (this=0x8501bb0, aName=0xbfffbef0, aResult=0x8501540)
    at /builds/trunk/mozilla/intl/strres/src/nsStringBundle.cpp:254
#20 0x427312ce in nsURLProperties::Get(nsAString const&, nsAString&) (
    this=0x8505a38, aKey=@0xbfffbee0, oValue=@0xbfffbfe0)
    at /builds/trunk/mozilla/intl/uconv/src/nsURLProperties.cpp:78
#21 0x42730ad2 in nsCharsetAlias2::GetPreferred(nsAString const&, nsAString&) (
    this=0x829a408, aAlias=@0x84f8728, oResult=@0xbfffbfe0)
    at /builds/trunk/mozilla/intl/uconv/src/nsCharsetAliasImp.cpp:107
#22 0x41f9a3cf in nsScanner::SetDocumentCharset(nsAString const&, int) (
    this=0x85028c8, aCharset=@0x84f8728, aSource=1)
    at /builds/trunk/mozilla/htmlparser/src/nsScanner.cpp:211
#23 0x41f99fad in nsScanner (this=0x85028c8, aFilename=@0xbfffc130,
    aCreateStream=0, aCharset=@0x84f8728, aSource=1)
    at /builds/trunk/mozilla/htmlparser/src/nsScanner.cpp:157
#24 0x41f90e7a in nsParser::Parse(nsIURI*, nsIRequestObserver*, int, void*,
nsDTDMode) (this=0x84f8678, aURL=0x82f4198, aListener=0x0, aVerifyEnabled=0,
    aKey=0x84d9a08, aMode=eDTDMode_autodetect)
    at /builds/trunk/mozilla/htmlparser/src/nsParser.cpp:1497
#25 0x416fcb19 in nsHTMLDocument::StartDocumentLoad(char const*, nsIChannel*,
nsILoadGroup*, nsISupports*, nsIStreamListener**, int, nsIContentSink*) (
    this=0x84d9a08, aCommand=0x4289c83a "view", aChannel=0x836af40,
    aLoadGroup=0x82f4208, aContainer=0x82f2f94, aDocListener=0xbfffceb0,
    aReset=1, aSink=0x0)
    at /builds/trunk/mozilla/content/html/document/src/nsHTMLDocument.cpp:1026
#26 0x413a93ce in nsContentDLF::CreateDocument(char const*, nsIChannel*,
nsILoadGroup*, nsISupports*, nsID const&, nsIStreamListener**, nsIContentViewer**) (
    this=0x84ae300, aCommand=0x4289c83a "view", aChannel=0x836af40,
    aLoadGroup=0x82f4208, aContainer=0x82f2f94, aDocumentCID=@0x41a1ed60,
    aDocListener=0xbfffceb0, aDocViewer=0xbfffcb30)
    at /builds/trunk/mozilla/layout/build/nsContentDLF.cpp:433
#27 0x413a8797 in nsContentDLF::CreateInstance(char const*, nsIChannel*,
nsILoadGroup*, char const*, nsISupports*, nsISupports*, nsIStreamListener**,
nsIContentViewer**) (this=0x84ae300, aCommand=0x4289c83a "view", aChannel=0x836af40,
    aLoadGroup=0x82f4208, aContentType=0xbfffcca0 "text/html",
    aContainer=0x82f2f94, aExtraInfo=0x0, aDocListener=0xbfffceb0,
    at /builds/trunk/mozilla/layout/build/nsContentDLF.cpp:233
#28 0x42866f36 in nsDocShell::NewContentViewerObj(char const*, nsIRequest*,
nsILoadGroup*, nsIStreamListener**, nsIContentViewer**) (this=0x82f2f70,
    aContentType=0xbfffcca0 "text/html", request=0x836af40,
    aLoadGroup=0x82f4208, aContentHandler=0xbfffceb0, aViewer=0xbfffcb30)
    at /builds/trunk/mozilla/docshell/base/nsDocShell.cpp:4551
#29 0x428667b3 in nsDocShell::CreateContentViewer(char const*, nsIRequest*,
nsIStreamListener**) (this=0x82f2f70, aContentType=0xbfffcca0 "text/html",
    request=0x836af40, aContentHandler=0xbfffceb0)
    at /builds/trunk/mozilla/docshell/base/nsDocShell.cpp:4422
#30 0x42894f52 in nsDSURIContentListener::DoContent(char const*, int,
nsIRequest*, nsIStreamListener**, int*) (this=0x82f3940,
    aContentType=0xbfffcca0 "text/html", aIsContentPreferred=0,
    request=0x836af40, aContentHandler=0xbfffceb0, aAbortProcess=0xbfffcd3c)
    at /builds/trunk/mozilla/docshell/base/nsDSURIContentListener.cpp:109
#31 0x41c96960 in nsDocumentOpenInfo::DispatchContent(nsIRequest*, nsISupports*)
(this=0x836a598, request=0x836af40, aCtxt=0x0)
    at /builds/trunk/mozilla/uriloader/base/nsURILoader.cpp:425
#32 0x41c95d70 in nsDocumentOpenInfo::OnStartRequest(nsIRequest*, nsISupports*)
    (this=0x836a598, request=0x836af40, aCtxt=0x0)
    at /builds/trunk/mozilla/uriloader/base/nsURILoader.cpp:228
#33 0x40ec2b38 in nsInputStreamChannel::OnStartRequest(nsIRequest*,
nsISupports*) (this=0x836af40, req=0x836b868, ctx=0x0)
    at /builds/trunk/mozilla/netwerk/base/src/nsInputStreamChannel.cpp:341
#34 0x40ec5d79 in nsInputStreamPump::OnStateStart() (this=0x836b868)
    at /builds/trunk/mozilla/netwerk/base/src/nsInputStreamPump.cpp:346
#35 0x40ec5c1b in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) (
    this=0x836b868, stream=0x836c4dc)
    at /builds/trunk/mozilla/netwerk/base/src/nsInputStreamPump.cpp:314
#36 0x40917355 in nsInputStreamReadyEvent::EventHandler(PLEvent*) (
    plevent=0x836d3e4) at /builds/trunk/mozilla/xpcom/io/nsStreamUtils.cpp:101
#37 0x4093cd82 in PL_HandleEvent (self=0x836d3e4)
    at /builds/trunk/mozilla/xpcom/threads/plevent.c:663
Proposed fixes:
 * better error handling in nsScanner::SetDocumentCharset
 * making nsCharsetAlias2::GetPreferred use the hardcoded defaults no matter what
 * (from bz) add .properties to the list of extensions we recognize internally
without hitting the user prefs
Attached patch fix #1 (diff -w)Splinter Review
This is one possibility, and I've tested that it works on its own.
Attached patch fix #2Splinter Review
This was bz's suggestion and I also tested that it fixes the bug on its own.
Comment on attachment 113662 [details] [diff] [review]
fix #2

I'd like to see what sort of perf impact this has (it may be a nice win on
Linux).... if none, we could back it out (I don't like forcing particular MIME
types on certain filenames...)
Attachment #113662 - Flags: superreview+
Attachment #113662 - Flags: review?(darin)
Comment on attachment 113661 [details] [diff] [review]
fix #1 (diff -w)

Simon?	What do you think about this one?

And we should probably clean up the nsScanner stuff too.... can we manage to
just not use it for XML?
Attachment #113661 - Flags: superreview+
Attachment #113661 - Flags: review?(smontagu)
Comment on attachment 113662 [details] [diff] [review]
fix #2

Attachment #113662 - Flags: review?(darin) → review+
This problem is very serious for Japanese.
But this bug is _not_ blocker of 1.3b.
If Moz1.3b has this problem,
We(mozilla-gumi) need to warn Japanese Mozilla users of this problem....
Blocks: 192171
Taking bug.
Assignee: heikki → dbaron
Whiteboard: [patch]
Target Milestone: --- → mozilla1.3beta
Comment on attachment 113662 [details] [diff] [review]
fix #2

(I have to say, I wish there were a way for the caller to force the MIME type,
since the caller knows it's a properties file, but the ".properties" extension
could, in theory, mean something else.)
Yeah, I have some plans to add the ability for callers to hint mime
types before calling Open/AsyncOpen... Those are still in the "plan"
stage, though.

Add an XXX comment here to remove this change once that plan goes through?
Comment on attachment 113661 [details] [diff] [review]
fix #1 (diff -w)

Attachment #113661 - Flags: review?(smontagu) → review+
Yeah, that would be fine with me as well...
Flags: blocking1.3b? → blocking1.3b-
Comment on attachment 113661 [details] [diff] [review]
fix #1 (diff -w)

Let's get this in for 1.3beta and our mozilla-japan testers if we can. If not
then we definitely want this for final.
Attachment #113661 - Flags: approval1.3b? → approval1.3b+
Fix (attachment 113661 [details] [diff] [review]) checked in to trunk, 2003-02-09 20:14 PST.

Filed bug 192522 on speeding up MIME type recognition for "*.properties" files
(the patch in attachment 113662 [details] [diff] [review]).
Closed: 17 years ago17 years ago
Resolution: --- → FIXED
I can't reproduce this problem with 2003021008-trunk/WinXP.
I'm very happy! because you fixed this bug in 1.3b.

Thank you!
*** Bug 192171 has been marked as a duplicate of this bug. ***
You need to log in before you can comment on or make changes to this bug.