52165 - Unable to access non-ASCII POP/Local mail folders

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P2)

Description

[ji]

*******observed with linux and win32 2000091108 build*****

Going to a Japanese POP or Local mail folder will get "unknown error".

Steps to reproduce:
1. Launch Mail.
2. Select File | New | Folder...
3. Select a POP account or Local Folders as the parent folder
4. Create a folder with Japanese name
5. Select the new created Japanese folder
You'll get a error msg window saying "unknown error"

If you click on OK to close the error msg window and go to the folder again,
you won't see the error msg, but the folder is not actually usable. 
(move/copying message to that folder won't work).

Comment 1

[ji]

There is no this problem with latin-1 mail folders,

Comment 2

[marina]

same thing happens with non-ascii name folders (french, german etc)

Comment 3

[marina]

it is a weird problem: it looks like the message itself is copied because the 
total ammount of the messages in this folder is changing but when you open it 
it doesn't display anything.

Comment 4

[ji]

On my Win95-J, the problem with latin-1 mail folders seems different with Ja 
folders. I don't get the "unknown error" when accessing the latin-1 folder and I 
can do move/copy msg between the latin-1 folders and the other folders. But 
after I restart the mail, the latin-1 mail folder name are not displayed well on 
the folder tree. The letters beginning from the accented character are stripped 
off, for example béx becomes b.

Comment 5

[marina]

i guess there is a problem with Latin-1 folder on JA system for Local mail as 
there is a problem with JA folder on US system, when i attempt to copy/move a 
message to a JA folder on the POP account in my US system i'll crash (this is a 
known problem)and we are not currently fixing it, right Naoki?

Comment 6

[ji]

Changed the platform to ALL.

Comment 7

[ji]

Nominating for nsbeta3.

Comment 8

[Katsuhiko Momoi]

QA contact to ji.
The problme is not observed with the 8/17/2000 Win32 build but
is observed with the 8/23 Win32 build.

Comment 9

[nhottanscp]

I can reproduce this on my WinNT JA.
Adding regression keyword. Adding kato san to cc, do you have time to look at 
this?

I saw two .msf are generated, one with a correct Japanese name other is 
something garbage.

Comment 10

[nhottanscp]

Attachment: screen shot, two .msf files generated for one Japanese folder

Comment 11

[nhottanscp]

P3 -> P2
I can reproduce this with Latin1 name using WinNT US. I will attach a test case.

Comment 12

[nhottanscp]

Attachment: zip file contains a mail folder "náoki"

Comment 13

[nhottanscp]

As I mentioned before there are two .msf files created.
For a folder "náoki", "náoki.msg" and "náoki.msf" are generated.
The second one is a file name encoded as UTF-8 which we do not support. We are 
using OS native charset for folder file name.

Alec made a change to nsParseMailbox.cpp (rev 1.119) on 8/10. The change assumes 
folder file name to be UTF-8. But Kat Momoi mentioned the problem not is 
observed with 8/17 build so may not be directly related.
Adding alecf to cc, do you know anything about this change? Do you know other 
place which assumes folder name to be UTF-8? I want to know the place which 
actually crates UTF-8 files.

Comment 14

[Frank Tang]

This is a new bug. We don't have this problem in 4x, neither in PR2. 
This is not a JA specific problem. This bug also impact any 8-bit charset, such 
as western european . 
Mark it as nsbeta3+ P2.

Comment 15

[ji]

Per Naoki's comments, it can be reproduced with latin-1 mail folder on US NT.
Changed the summary accordingly.

Comment 16

[nhottanscp]

>For a folder "náoki", "náoki.msg" and "náoki.msf" are generated.
typo, should be both ".msf".
For a folder "náoki", "náoki.msf" and "náoki.msf" are generated.

I also forgot to cc alecf.

Comment 17

[nhottanscp]

Xianglan found that the problem is not in 8/23 build but started with 8/24 
build.

Comment 18

[nhottanscp]

I checked Bonsai and found there were checkins for DOM string changes.
I think RDF behavior was changed by this. Here is one of the changes.
rdf/content/src/nsXULElement.cpp, rev=1.292
Before change:
nsCAutoString tmp; tmp.AssignWithConversion(aName);
After change:
nsCAutoString tmp; tmp.Assign(NS_ConvertUCS2toUTF8(aName));           

This might had affected local folder. I will investigate more tomorrow.

Comment 19

[nhottanscp]

I think I found the place which sets UTF-8 path name.
RDFServiceImpl::GetUnicodeResource creates UTF-8 path name.
So nsMsgLocalMailFolder::Init is called in different ways (see two call stacks 
below). The second case, I understand that it's adding subfolders. But the first 
case I don't know what it is doing.
Adding waterson to cc, do you have any idea about what it's doing for the first 
call stack? The break point is after I click a local mail folder.

call stack 1:
nsMsgLocalMailFolder::Init(nsMsgLocalMailFolder * const 0x04014460, const char * 
0x0012b2a4) line 468
RDFServiceImpl::GetResource(RDFServiceImpl * const 0x00b6ace0, const char * 
0x0012b2a4, nsIRDFResource * * 0x04014760) line 761 + 16 bytes
RDFServiceImpl::GetUnicodeResource(RDFServiceImpl * const 0x00b6ace0, const 
unsigned short * 0x0012b324, nsIRDFResource * * 0x04014760) line 779 + 38 bytes
nsXULElement::GetResource(nsXULElement * const 0x040147a4, nsIRDFResource * * 
0x04014760) line 3526 + 34 bytes
nsXULElement::ForceElementToOwnResource(nsXULElement * const 0x040147a0, int 
0x00000001) line 1703 + 49 bytes
nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x038c5b70, 
nsIContent * 0x03cf9640, nsIContent * 0x03fd01c0, int 0x00000001, nsIRDFResource 
* 0x03cfa100, int 0x00000000, Match * 0x03290eb8, nsIContent * * 0x0012caf0, int 
* 0x0012caf4) line 5418
nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x038c5c80, 
nsIContent * 0x03cf9640, nsIContent * 0x03cf9640, int 0x00000001, nsIRDFResource 
* 0x03cfa100, int 0x00000000, Match * 0x03290eb8, nsIContent * * 0x0012caf0, int 
* 0x0012caf4) line 5374 + 61 bytes
nsXULTemplateBuilder::CreateContainerContents(nsIContent * 0x03cf9640, 
nsIRDFResource * 0x03cd56f0, int 0x00000000, nsIContent * * 0x0012caf0, int * 
0x0012caf4) line 6145
nsXULTemplateBuilder::OpenContainer(nsXULTemplateBuilder * const 0x03b15924, 
nsIContent * 0x03cf9640) line 4317 + 54 bytes
nsXULDocument::OpenWidgetItem(nsIContent * 0x03cf9640) line 4488 + 22 bytes
nsXULDocument::AttributeChanged(nsXULDocument * const 0x03763070, nsIContent * 
0x03cf9640, int 0x00000000, nsIAtom * 0x01a0edd0, int 0xffffffff) line 1645
nsXULElement::SetAttribute(nsXULElement * const 0x03cf9640, nsINodeInfo * 
0x03fd0ba0, const basic_nsAReadableString<unsigned short> & {...}, int 
0x00000001) line 2759
nsXULElement::SetAttribute(nsXULElement * const 0x03cf9644, const 
basic_nsAReadableString<unsigned short> & {...}, const 
basic_nsAReadableString<unsigned short> & {...}) line 1229 + 31 bytes
ElementSetAttribute(JSContext * 0x03740a30, JSObject * 0x032958d8, unsigned int 
0x00000002, long * 0x0323df3c, long * 0x0012d1d4) line 239 + 26 bytes
js_Invoke(JSContext * 0x03740a30, unsigned int 0x00000002, unsigned int 
0x00000000) line 731 + 23 bytes
js_Interpret(JSContext * 0x03740a30, long * 0x0012db5c) line 2538 + 15 bytes
js_Invoke(JSContext * 0x03740a30, unsigned int 0x00000001, unsigned int 
0x00000002) line 748 + 13 bytes
js_InternalInvoke(JSContext * 0x03740a30, JSObject * 0x032ba198, long 
0x032ba720, unsigned int 0x00000000, unsigned int 0x00000001, long * 0x0012dcf0, 
long * 0x0012dc80) line 821 + 19 bytes
JS_CallFunctionValue(JSContext * 0x03740a30, JSObject * 0x032ba198, long 
0x032ba720, unsigned int 0x00000001, long * 0x0012dcf0, long * 0x0012dc80) line 
3175 + 31 bytes
nsJSContext::CallEventHandler(nsJSContext * const 0x03741130, void * 0x032ba198, 
void * 0x032ba720, unsigned int 0x00000001, void * 0x0012dcf0, int * 0x0012dcec, 
int 0x00000000) line 906 + 33 bytes
nsJSEventListener::HandleEvent(nsIDOMEvent * 0x03cedcf4) line 154 + 64 bytes
nsXBLEventHandler::ExecuteHandler(nsXBLEventHandler * const 0x03cfeef0, const 
basic_nsAReadableString<unsigned short> & {...}, nsIDOMEvent * 0x03cedcf4) line 
593
nsXBLEventHandler::MouseClick(nsIDOMEvent * 0x03cedcf4) line 298 + 36 bytes
nsEventListenerManager::HandleEvent(nsIPresContext * 0x03765680, nsEvent * 
0x0012f48c, nsIDOMEvent * * 0x0012f3b0, nsIDOMEventTarget * 0x03c390e8, unsigned 
int 0x00000002, nsEventStatus * 0x0012f784) line 861 + 23 bytes
nsXULElement::HandleDOMEvent(nsXULElement * const 0x03c390e0, nsIPresContext * 
0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 
0x00000002, nsEventStatus * 0x0012f784) line 3307
nsXULElement::HandleDOMEvent(nsXULElement * const 0x03cf9640, nsIPresContext * 
0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 
0x00000002, nsEventStatus * 0x0012f784) line 3324 + 39 bytes
nsXULElement::HandleDOMEvent(nsXULElement * const 0x03a8bdb0, nsIPresContext * 
0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 
0x00000002, nsEventStatus * 0x0012f784) line 3324 + 39 bytes
nsXULElement::HandleDOMEvent(nsXULElement * const 0x03fa4d10, nsIPresContext * 
0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 
0x00000002, nsEventStatus * 0x0012f784) line 3324 + 39 bytes
nsXULElement::HandleDOMEvent(nsXULElement * const 0x03fa52f0, nsIPresContext * 
0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 
0x00000001, nsEventStatus * 0x0012f784) line 3324 + 39 bytes
PresShell::HandleEventInternal(nsEvent * 0x0012f48c, nsIView * 0x00000000, 
nsEventStatus * 0x0012f784) line 4182 + 45 bytes
PresShell::HandleEventWithTarget(PresShell * const 0x03766ab0, nsEvent * 
0x0012f48c, nsIFrame * 0x03279130, nsIContent * 0x03fa52f0, nsEventStatus * 
0x0012f784) line 4163 + 18 bytes
nsEventStateManager::CheckForAndDispatchClick(nsEventStateManager * const 
0x03b94080, nsIPresContext * 0x03765680, nsMouseEvent * 0x0012f894, 
nsEventStatus * 0x0012f784) line 1820 + 59 bytes
nsEventStateManager::PostHandleEvent(nsEventStateManager * const 0x03b94088, 
nsIPresContext * 0x03765680, nsEvent * 0x0012f894, nsIFrame * 0x03279130, 
nsEventStatus * 0x0012f784, nsIView * 0x03d0b330) line 901 + 28 bytes
PresShell::HandleEventInternal(nsEvent * 0x0012f894, nsIView * 0x03d0b330, 
nsEventStatus * 0x0012f784) line 4202 + 43 bytes
PresShell::HandleEvent(PresShell * const 0x03766ab4, nsIView * 0x03d0b330, 
nsGUIEvent * 0x0012f894, nsEventStatus * 0x0012f784, int 0x00000001, int & 
0x00000001) line 4117 + 23 bytes
nsView::HandleEvent(nsView * const 0x03d0b330, nsGUIEvent * 0x0012f894, unsigned 
int 0x0000001c, nsEventStatus * 0x0012f784, int 0x00000001, int & 0x00000001) 
line 379
nsViewManager2::DispatchEvent(nsViewManager2 * const 0x03765330, nsGUIEvent * 
0x0012f894, nsEventStatus * 0x0012f784) line 1429
HandleEvent(nsGUIEvent * 0x0012f894) line 68
nsWindow::DispatchEvent(nsWindow * const 0x03d0bb14, nsGUIEvent * 0x0012f894, 
nsEventStatus & nsEventStatus_eIgnore) line 614 + 10 bytes
nsWindow::DispatchWindowEvent(nsGUIEvent * 0x0012f894) line 635
nsWindow::DispatchMouseEvent(unsigned int 0x0000012d, nsPoint * 0x00000000) line 
3811 + 21 bytes
ChildWindow::DispatchMouseEvent(unsigned int 0x0000012d, nsPoint * 0x00000000) 
line 4021
nsWindow::ProcessMessage(unsigned int 0x00000202, unsigned int 0x00000000, long 
0x001b0007, long * 0x0012fc10) line 2889 + 24 bytes
nsWindow::WindowProc(HWND__ * 0x006d07b4, unsigned int 0x00000202, unsigned int 
0x00000000, long 0x001b0007) line 883 + 27 bytes

call stack 2:
nsMsgLocalMailFolder::Init(nsMsgLocalMailFolder * const 0x03cfa100, const char * 
0x0012c118) line 468
RDFServiceImpl::GetResource(RDFServiceImpl * const 0x00b6ace0, const char * 
0x0012c118, nsIRDFResource * * 0x0012c0f0) line 761 + 16 bytes
nsMsgLocalMailFolder::AddSubfolder(nsMsgLocalMailFolder * const 0x03cd570c, 
nsAutoString * 0x0012c1bc, nsIMsgFolder * * 0x0012c1b4) line 530 + 69 bytes
nsMsgLocalMailFolder::CreateSubFolders(nsFileSpec & {...}) line 490 + 62 bytes
nsMsgLocalMailFolder::GetSubFolders(nsMsgLocalMailFolder * const 0x03cd570c, 
nsIEnumerator * * 0x0012c328) line 705 + 15 bytes
nsMsgFolderDataSource::createFolderChildNode(nsIMsgFolder * 0x03cd570c, 
nsIRDFNode * * 0x0012c3e4) line 1468 + 36 bytes
nsMsgFolderDataSource::createFolderNode(nsIMsgFolder * 0x03cd570c, 
nsIRDFResource * 0x0229d390, nsIRDFNode * * 0x0012c3e4) line 900 + 16 bytes
nsMsgFolderDataSource::GetTarget(nsMsgFolderDataSource * const 0x03cce280, 
nsIRDFResource * 0x03cd56f0, nsIRDFResource * 0x0229d390, int 0x00000001, 
nsIRDFNode * * 0x0012c3e4) line 305 + 25 bytes
CompositeDataSourceImpl::GetTarget(CompositeDataSourceImpl * const 0x03b15150, 
nsIRDFResource * 0x03cd56f0, nsIRDFResource * 0x0229d390, int 0x00000001, 
nsIRDFNode * * 0x0012c3e4) line 721 + 28 bytes
nsXULTemplateBuilder::CheckContainer(nsIRDFResource * 0x03cd56f0, int * 
0x0012cb20, int * 0x0012cb24) line 6284 + 65 bytes
nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x038c5b70, 
nsIContent * 0x03888fa0, nsIContent * 0x03c390e0, int 0x00000001, nsIRDFResource 
* 0x03cd56f0, int 0x00000000, Match * 0x03290b00, nsIContent * * 0x0012dc18, int 
* 0x0012dc1c) line 5423 + 29 bytes
nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x038c5c80, 
nsIContent * 0x03888fa0, nsIContent * 0x03888fa0, int 0x00000001, nsIRDFResource 
* 0x03cd56f0, int 0x00000000, Match * 0x03290b00, nsIContent * * 0x0012dc18, int 
* 0x0012dc1c) line 5374 + 61 bytes
nsXULTemplateBuilder::CreateContainerContents(nsIContent * 0x03888fa0, 
nsIRDFResource * 0x03ccebb0, int 0x00000000, nsIContent * * 0x0012dc18, int * 
0x0012dc1c) line 6145
nsXULTemplateBuilder::CreateTemplateAndContainerContents(nsIContent * 
0x03888fa0, nsIContent * * 0x0012dc18, int * 0x0012dc1c) line 6038 + 34 bytes
nsXULTemplateBuilder::RebuildContainerInternal(nsIContent * 0x03888fa0, int 
0x00000000) line 4476 + 40 bytes
nsXULTemplateBuilder::RebuildContainer(nsXULTemplateBuilder * const 0x03b15924, 
nsIContent * 0x03888fa0) line 4433 + 17 bytes
nsXULDocument::RebuildWidgetItem(nsIContent * 0x03888fa0) line 4591 + 22 bytes
nsXULDocument::AttributeChanged(nsXULDocument * const 0x03763070, nsIContent * 
0x03888fa0, int 0x00000000, nsIAtom * 0x025cad50, int 0xffffffff) line 1671
nsXULElement::SetAttribute(nsXULElement * const 0x03888fa0, nsINodeInfo * 
0x03b53260, const basic_nsAReadableString<unsigned short> & {...}, int 
0x00000001) line 2759
nsXULElement::SetAttribute(nsXULElement * const 0x03888fa4, const 
basic_nsAReadableString<unsigned short> & {...}, const 
basic_nsAReadableString<unsigned short> & {...}) line 1229 + 31 bytes
nsXULTreeElement::SetAttribute(nsXULTreeElement * const 0x03c9ca28, const 
basic_nsAReadableString<unsigned short> & {...}, const 
basic_nsAReadableString<unsigned short> & {...}) line 54 + 29 bytes
ElementSetAttribute(JSContext * 0x03740a30, JSObject * 0x00f31f60, unsigned int 
0x00000002, long * 0x03279174, long * 0x0012e344) line 239 + 26 bytes
js_Invoke(JSContext * 0x03740a30, unsigned int 0x00000002, unsigned int 
0x00000000) line 731 + 23 bytes
js_Interpret(JSContext * 0x03740a30, long * 0x0012eccc) line 2538 + 15 bytes
js_Invoke(JSContext * 0x03740a30, unsigned int 0x00000001, unsigned int 
0x00000002) line 748 + 13 bytes
js_InternalInvoke(JSContext * 0x03740a30, JSObject * 0x00f55628, long 
0x00d202a8, unsigned int 0x00000000, unsigned int 0x00000001, long * 0x0012ee60, 
long * 0x0012edf0) line 821 + 19 bytes
JS_CallFunctionValue(JSContext * 0x03740a30, JSObject * 0x00f55628, long 
0x00d202a8, unsigned int 0x00000001, long * 0x0012ee60, long * 0x0012edf0) line 
3175 + 31 bytes
nsJSContext::CallEventHandler(nsJSContext * const 0x03741130, void * 0x00f55628, 
void * 0x00d202a8, unsigned int 0x00000001, void * 0x0012ee60, int * 0x0012ee5c, 
int 0x00000000) line 906 + 33 bytes
nsJSEventListener::HandleEvent(nsIDOMEvent * 0x03c5b054) line 154 + 64 bytes
nsEventListenerManager::HandleEventSubType(nsListenerStruct * 0x03773350, 
nsIDOMEvent * 0x03c5b054, nsIDOMEventTarget * 0x037411b0, unsigned int 
0x00000001, unsigned int 0x00000007) line 788 + 19 bytes
nsEventListenerManager::HandleEvent(nsIPresContext * 0x03765680, nsEvent * 
0x0012f3b0, nsIDOMEvent * * 0x0012f374, nsIDOMEventTarget * 0x037411b0, unsigned 
int 0x00000007, nsEventStatus * 0x0012f3d4) line 1361 + 39 bytes
GlobalWindowImpl::HandleDOMEvent(GlobalWindowImpl * const 0x037411a0, 
nsIPresContext * 0x03765680, nsEvent * 0x0012f3b0, nsIDOMEvent * * 0x0012f374, 
unsigned int 0x00000001, nsEventStatus * 0x0012f3d4) line 467
DocumentViewerImpl::LoadComplete(DocumentViewerImpl * const 0x03764c00, unsigned 
int 0x00000000) line 668 + 47 bytes
nsWebShell::OnEndDocumentLoad(nsWebShell * const 0x037418d8, nsIDocumentLoader * 
0x037416b0, nsIChannel * 0x02e3ae10, unsigned int 0x00000000) line 981
nsDocLoaderImpl::FireOnEndDocumentLoad(nsDocLoaderImpl * 0x037416b0, nsIChannel 
* 0x02e3ae10, unsigned int 0x00000000) line 804
nsDocLoaderImpl::DocLoaderIsEmpty(unsigned int 0x00000000) line 611
nsDocLoaderImpl::OnStopRequest(nsDocLoaderImpl * const 0x037416b4, nsIChannel * 
0x03c5f7c0, nsISupports * 0x00000000, unsigned int 0x00000000, const unsigned 
short * 0x100a56b0 gCommonEmptyBuffer) line 554
nsLoadGroup::RemoveChannel(nsLoadGroup * const 0x03741650, nsIChannel * 
0x03c5f7c0, nsISupports * 0x00000000, unsigned int 0x00000000, const unsigned 
short * 0x100a56b0 gCommonEmptyBuffer) line 566 + 39 bytes
nsFileChannel::OnStopRequest(nsFileChannel * const 0x03c5f7c8, nsIChannel * 
0x03c5f4d0, nsISupports * 0x00000000, unsigned int 0x00000000, const unsigned 
short * 0x100a56b0 gCommonEmptyBuffer) line 656
nsOnStopRequestEvent::HandleEvent(nsOnStopRequestEvent * const 0x03ca4030) line 
302
nsStreamListenerEvent::HandlePLEvent(PLEvent * 0x03ca50e0) line 97 + 12 bytes
PL_HandleEvent(PLEvent * 0x03ca50e0) line 575 + 10 bytes
PL_ProcessPendingEvents(PLEventQueue * 0x00b46840) line 508 + 9 bytes
_md_EventReceiverProc(HWND__ * 0x00440676, unsigned int 0x0000c102, unsigned int 
0x00000000, long 0x00b46840) line 1044 + 9 bytes
USER32! 77e71820()

Comment 20

[nhottanscp]

I think RDF to use UTF-8 URI is the right thing.
So my current plan is to change nsMsgLocalMailFolder to pass UTF-8 URI to RDF 
instead of OS native charset. Then convert the UTF-8 string to OS native charset  
for the DB construction.

1) nsMsgLocalMailFolder::AddSubfolder,
Change to call GetResource() with UTF-8 URI.

2) nsMsgLocalMailFolder::Init,
Add a conversion from UTF-8 to OS native charset before calling 
nsMsgDBFolder::Init.

I made those changes to my local tree. But nsMsgLocalMailFolder::Init is still 
called with OS native charset in few cases. I need to investigate those cases 
and change to use UTF-8.

Comment 21

[Alec Flett]

ugh - this basically means any place where we call "GetResource" where the 
folder hasn't been created yet. this might come from preferences like the 
Send/Drafts/etc folders

Comment 22

[nhottanscp]

>2) nsMsgLocalMailFolder::Init,
This was wrong. I should change nsMsgFolder::Init() instead.
It calls nsRDFResource::Init() and CreateBaseMessageURI().
So I passed UTF-8 to nsRDFResource::Init() and OS charset to 
CreateBaseMessageURI().

I did above change. Now I am getting back double-converted UTF-8 from RDF.
For 'á' I am getting 0xC383C2A1 instead of 0xC3A1.
This usually happens when using AssignWithConversion() to convert 8 bit data to 
unicode then use NS_ConvertUCS2toUTF8 to create UTF-8 string.
I feel that RDF is not always expecting UTF-8 URI.
A question to waterson: Is nsRDFResource::Init() expecting a URI as UTF-8? If 
not, is there other interface with takes UTF-8 URI.

Part of the call stack which passing a corrupted UTF-8 URI:
nsMsgLocalMailFolder::Init(nsMsgLocalMailFolder * const 0x03d6e9d0, const char * 
0x0012b2a4) line 479
RDFServiceImpl::GetResource(RDFServiceImpl * const 0x00b6a4a0, const char * 
0x0012b2a4, nsIRDFResource * * 0x03d6ec90) line 761 + 16 bytes
RDFServiceImpl::GetUnicodeResource(RDFServiceImpl * const 0x00b6a4a0, const 
unsigned short * 0x0012b324, nsIRDFResource * * 0x03d6ec90) line 779 + 38 bytes
nsXULElement::GetResource(nsXULElement * const 0x03d6ecd4, nsIRDFResource * * 
0x03d6ec90) line 3526 + 34 bytes
nsXULElement::ForceElementToOwnResource(nsXULElement * const 0x03d6ecd0, int 
0x00000001) line 1703 + 49 bytes
nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x03986b30, 
nsIContent * 0x03d5aa20, nsIContent * 0x03d6fbb0, int 0x00000001, nsIRDFResource 
* 0x03d5ba30, int 0x00000000, Match * 0x03567580, nsIContent * * 0x0012caf0, int 
* 0x0012caf4) line 5418
nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x03986c40, 
nsIContent * 0x03d5aa20, nsIContent * 0x03d5aa20, int 0x00000001, nsIRDFResource 
* 0x03d5ba30, int 0x00000000, Match * 0x03567580, nsIContent * * 0x0012caf0, int 
* 0x0012caf4) line 5374 + 61 bytes

Comment 23

[Chris Waterson]

> A question to waterson: Is nsRDFResource::Init() expecting a URI as UTF-8?

Yes. It will store the bytes you give to it.

Comment 24

[Alec Flett]

Yes!! I am seeing this double conversion in the filters as well. Something is 
screwy in RDF...

Also, I'm concerned about you changing CreateBaseMessageURI to use the platform 
charset - there is other code (namely nsMessage::GetMsgFolder()) which assumes 
that the base message URI is a direct copy of the folder URI, with some 
modification. The message and folder URIs should be in the same charset no 
matter what.

I have a feeling that this double-conversion of UTF8 is the real source of the 
problem, as this USED to work.

Comment 25

[Chris Waterson]

The only conversion that core RDF does (modulo parsing RDF/XML) is in
nsRDFService::GetUnicodeResource(). In this case, RDF is given a UCS-2 string,
and it uses NS_ConvertUCS2toUTF8() to deflate it to a single-byte representation.

nsRDFResource::Init() accepts a single-byte string, and is agnostic as to its
representation. If you've created the resource "properly" (i.e., using
nsRDFService), then it will contain a UTF-8 encoded string. If you create the
RDF resource using your own mechanism, it's up to you to ensure that you play
nice with the bytes.

Comment 26

[nhottanscp]

>Also, I'm concerned about you changing CreateBaseMessageURI to use the platform 
>charset
This is not a change, it's been using the platform charset. My change is to pass 
UTF-8 to nsRDFResource::Init().

Comment 27

[nhottanscp]

>If you've created the resource "properly"
So does this mean creating RDF without using nsRDFResource::Init()?
As I am not familiar with RDF (also local mailfodler code), I dont' know what 
kind of change requred.
Would it affects a lot to local mailfolder code?

Comment 28

[Chris Waterson]

The "normal" way of getting at nsRDFResource objects is to call
nsIRDFService::Get[Unicode]Resource(). *This* code does the work of creating an
nsRDFResource object, and then calling its Init() method with a string. AFAIK,
this is the *only* was that nsRDFResource objects get created, but I may be
wrong wrt mailnews. alecf?

Now, I mis-spoke a bit, above. If you call nsIRDFService::GetResource(), you'll
be handing in a "naked" single-byte string, and RDF won't know how it's encoded.
So it'll just take the bytes and pass them along.

Comment 29

[Alec Flett]

right, mailnews doesn't do anything special to create resources, we always just 
call GetResource

But my point is that CreateBaseMessageURI should use exactly the same charset as 
the folder. whatever nsMsgFolder::Init() gets, it should pass this on to 
CreateBaseMessageURI - there should be no conversion.

Comment 30

[nhottanscp]

I agree, the URI should also contain UTF-8 string. The problem is that the URI 
is tightly connected with actual file name. Since we cannot use UTF-8 as a real 
file name, we need to find a place where the URI is used to get a file name then 
apply the conversion there.

But above issue is not related to the double-conversion problem.
In the partial call stack I put today, nsXULElement::GetResource calling 
GetUnicodeResource so nsMsgLocalMailFolder::Init is getting double-converted 
UTF-8 URI. nsMsgLocalMailFolder cannot control GetResource or GetUnicodeResource 
is used.

Comment 31

[Alec Flett]

ooh! This totally explains what's going on! Oh man. This is not a mail problem, 
this is a XUL template or nsXULElement problem.

Here's what's happening: the template builder is probably saving the direct 
ASCII version of the URI of the resource in the nsXULElement that it creates. 
This gets converted to unicode (i.e. a direct conversion, no charsets involved) 
when the attribute is read.

Then, the template builder is calling GetUnicodeResource which means the UTF8 
string, which is currently a UTF8 string stored in PRUnichar format, gets 
re-encoded in UTF8 by GetUnicodeResource.

This is really bad.
I think that either GetUnicodeResource has to do an ASCII conversion instead of 
UTF8, for nsbeta3. After nsbeta3 we need to fix the template/XUL code to do the 
right conversions in the right places.

this also explains bug 52190, because the filter datasource is getting queried 
on a string that has been double-converted into UTF8

Comment 32

[Chris Waterson]

With respect to nsIRDFService::GetUnicodeResource(), we can't change it, because
that would undo several nsbeta2 bug fixes. Sorry! :-(

I also think that nsXULElement::GetResource() is implemented correctly.

That said, I could believe that somebody isn't properly *encoding* to UCS-2
before setting the attribute's value...I gotta run, but maybe poke around in
nsXULContentUtils.cpp (which is what nsXULTemplateBuilder relies on to do a lot
of the URI-to-attribute value transformation).

Comment 33

[Alec Flett]

ugh! ok. not surprised. I'll poke around at RDF and see what I can find

adding robert, the other template wizard.
robert - read my previous comment or two

Comment 34

[nhottanscp]

I found one place in RDF which applies AssignWithConversion to uri.
This creates a nsString with a data actually contains UTF-8. Later if we do 
NS_ConvertUCS2toUTF8 then we get the double-converted string.

nsXULTemplateBuilder.cpp
5155 nsXULTemplateBuilder::BuildContentFromTemplate

5410             nsAutoString id; id.AssignWithConversion(uri);

There are two other places which are related to this bug does 
AssignWithConversion.
They may be okay, as long as we convert the string to OS charset before doing 
it.
nsMsgFolder.cpp
525 nsMsgFolder::parseURI(PRBool needServer)

583       mName.AssignWithConversion(result);

nsLocalFolderSummarySpec.cpp
 65 void nsLocalFolderSummarySpec:: CreateSummaryFileName()

 71     nsString fullLeafName; fullLeafName.AssignWithConversion(leafName);

Comment 35

[Alec Flett]

the RDF one is exactly it! it's the ID that we're concerned about, so this is 
probably it. Anyway, try converting that one to convert from UTF8 to UCS2, and 
see if that makes the problem go away.

the assignment to mName may also have something to do with it, we should 
probably be converting from UTF8 there too.

I have no idea about the third case, but I would guess that we should fix it 
there too.

I think the RDF one is the most important though. if that fixes the problem, I 
say we should just fix that.

Comment 36

[nhottanscp]

After I changed nsXULTemplateBuilder::BuildContentFromTemplate to use 
NS_ConvertUTF8toUCS2 instead of AssignWithConversion,
looks like the double conversion problem has gone.

But looks like I need to take care other things too. I am getting UTF-8 string 
for CreateSummaryFileName and a UTF-8 .msf is created.

Comment 37

[Chris Waterson]

Yep, nsXULTemplateBuilder.cpp:4949 looks suspect also...

Comment 38

[Alec Flett]

Naoki, can you attach a patch here, and we can all try it out? Chris's other
finding would be good too.

Comment 39

[Chris Waterson]

Attachment: untested patch along the lines of what nhotta suggested

Comment 40

[nhottanscp]

I applied the patch but still does not work.
This bug was introduced on 8/23. I think that's because RDF started to return 
UTF-8 URI occasionally which I think related to the DOM string change.
So the bug cannot be fixed as long as local mail depends on OS charset.

Comment 41

[nhottanscp]

Local mail needs to use file names as OS charset even URI is UTF-8.
So I need help from mailnew eng. to identify the place(s) to put a conversion 
between UTF-8 and OS charset.

Comment 42

[scottputterman]

Naoki and I did some debugging and this is what I'm seeing:

We are converting the uri to UTF8 for each folder. When we call
NotifyFolderEvent we are passing in this uri.

However, before we can do anything like change folders we first check to see if
the uri passed into NotifyFolderEvent is the same as the uri we saved off(we
save off the uri to make sure we don't do anything for a folder that isn't
loaded into the current window).

We get the saved off uri from getAttribute on the selected folder tree item.
We do this by doing uri = selectedFolder.getAttribute("id"). This does not
appear to be coming back in UTF8 and hence our comparisons when the folder is
loaded, as mentioned above, fails.

So, my question is, should we be using something other than getAttribute to get
back the UTF8 uri we stick into the resource?

Comment 43

[scottputterman]

ok, I just talked to Naoki. It seems that what I wrote was wrong.  GetAttribute
is returning the correct value but we are not sending the correct value to
NotifyFolderEvent.  We are looking in to that.  Unless we change our minds
again, ignore my previous comments (except for that fact that the 2 strings
aren't equal which seems to be the problem).

Comment 44

[Michael La Guardia]

pdt agrees p2

Comment 45

[nhottanscp]

It was hard to check in the console which one was the correct UTF-8 string.
But it turns out that the Scott's original comment is right. Sorry about the 
confusion.
We are sending a correct UTF-8 to NotifyFolderEvent (0xC3A1 for á). But from 
getAttribute() we get non UTF-8 value (0xE1 for á).

Comment 46

[Alec Flett]

I just mentioned this to scott - you could also try accessing the "id" attribute
as in folder.id instead of folder.getAttribute("id"); - it goes through a
slightly different path in the DOM code.

Comment 47

[nhottanscp]

I tried folder.id to get URI but the same result. It did not return UTF-8 URI.

Comment 48

[nhottanscp]

So the current issue is that there are two URI in JS code which do not match.
The uri passed into NotifyFolderEvent is UTF-8.
And, uri = selectedFolder.getAttribute("id"), this is UCS2.

In JS file, I think URI should not be raw UTF-8. Is there way to get the URI in 
UCS2 in JS?

Comment 49

[Alec Flett]

if it's not doing that, it's a bug. the "id" attribute is supposed to return the 
UCS2 representation of the attribute, from JS. if it's not it's a bug where 
someone is using AssignWithConversion (or one of the other 
AppendWithConversion/etc) operators incorrectly.

Comment 50

[nhottanscp]

selectedFolder.getAttribute is returning UCS2. The other one returns UTF-8 
that's why the two do not match.

Comment 51

[nhottanscp]

Currently, I am blocked by the above issue (URI in UCS2 and UTF8 not match), any 
help would be appreciated.

In the JS file, I hacked the JS code to proceed. Then I can see messages in 
thread pane.
msgMail3PaneWindow.js
uri = gCurrentFolderToReroot;
if(uri == gCurrentFolderToReroot)

Also changed nsMailboxService::FetchMessage which is getting a non UTF-8 URI. So 
I put a conversion from OS charset to UTF-8 there (not sure who is passing OS 
charset and if this is local mail specific or not).
Then I can see a message, finally.

Comment 52

[Chris Waterson]

can you attach a patch with all of the relevant changes?

Comment 53

[scottputterman]

what you are saying is that we need nsRDFResource::Value to be in UCS2? Right
now it's just returning what we've stored in it, which is UTF8.

Comment 54

[nhottanscp]

Yes, that UTF-8 URI doesn't match with the URI by getAttribute.

Comment 55

[nhottanscp]

Attachment: patch 1: JS changes to use GetUnicodeResource

Comment 56

[nhottanscp]

Attachment: patch 2: charset conversion in mailnews/base/util

Comment 57

[nhottanscp]

Attachment: patch 3: charset conversion in mailnews/local/src

Comment 58

[nhottanscp]

I attached the changes in my local tree so far. I also has the waterson's patch 
previously attached. 
The bug is reproducible on US system with the previously attached mailbox data.

Comment 59

[nhottanscp]

I applied the patches so far and tested on Japanese system.
I tried a Japanese local mail folder. The thread pane displays messages fine.
But when I clicked a message, I got two asserts then the "Unknown error" dialog 
shows up. I will attach the call stack.

The first one is inXPCConvert::JSData2Native, xpcconvert.cpp.
NS_ASSERTION(legalRange,"U+0080/U+0100 - U+FFFF data lost");
The string is a valid unicode (japanese hiragana). But failing because of the 
following check.

//#define STRICT_CHECK_OF_UNICODE
#ifdef STRICT_CHECK_OF_UNICODE
#define ILLEGAL_RANGE(c) (0!=((c) & 0xFF80))
#else // STRICT_CHECK_OF_UNICODE
#define ILLEGAL_RANGE(c) (0!=((c) & 0xFF00))
#endif // STRICT_CHECK_OF_UNICODE


The second one is in nsMailboxProtocol::SetupMessageExtraction, in 
nsMailboxProtocol.cpp.
rv = m_runningUrl->GetMessageHeader(getter_AddRefs(msgHdr));
This failed, I think due to an invalid URL.

Comment 60

[nhottanscp]

Attachment: the two call stacks I mentioned in my last comment

Comment 61

[nhottanscp]

In xpcconvert.cpp, by defining STRICT_CHECK_OF_UNICODE, I can also activate 
the assersion for NativeData2JS.

I hit the assersion using the Japanese folder name. It's passing the japanese 
folder name in UTF-8. But this routine does not expect UTF-8 as an input. The JS 
caller was nsMsgMail3PaneWindow.js line 102.
102                                 var uri = resource.Value;

That's the same place we got the problem of not matching UTF-8 and UCS2 URI.

Comment 62

[nhottanscp]

I added GetUnicodeValue() to nsIRDFResource and used it in the JS file. That 
seems to solve the URI encoding inconsistency problem.
I have no clue about the Japanese folder name problem. I will apply the 
nsIRDFResource change to my Japanese build and see if that can change anything.

Comment 63

[nhottanscp]

Attachment: patch 4: RDF change to add GetUnicodeValue

Comment 64

[nhottanscp]

Attachment: patch 5: JS change to use GetUnicodeValue, replacing patch 1

Comment 65

[Alec Flett]

it sounds like this is the right thing - otherwise there's no way to get the 
UCS2 version of the string from an RDF Resource.
(as a side note, I believe you can say 
*aURI = NS_ConvertUTF8toUCS2(mURI).ToNewUnicode()
to avoid an extra strcpy)

Comment 66

[nhottanscp]

For Japanese URI problem, the URI is UCS2 but 
nsIMessenger::OpenURL(in string url), this is a char* interface.
So all the Japanese characters were stripped out (e.g. 0x3042 => 42).
Latin1 is fine since upper byte is zero (e.g. 00E1 => E1).

So either need to add unicode version of OpenURL or escape the folder name part.

Comment 67

[nhottanscp]

I changed to escape the UTF-8 string in AddSubfolder before passing to RDF.
This works fine since our code works when the data is 7 bit (that's why IMAP is 
working fine with japanese folder name by using UTF-7), thanks to Frank for the 
suggestion.
So we do not need to change any JS file and RDF. Just need escape/unesape and 
charset conversions (native charset and UTF-8).
I will attach a diff for review.

Comment 68

[nhottanscp]

Attachment: patch for review

Comment 69

[Alec Flett]

good job tracking this stuff down naoki. I'll try to finish reviewing these 
thursday morning so you can checkin.

Comment 70

[nhottanscp]

Message delete does not work with the patch. I am investigating.

Comment 71

[nhottanscp]

Message delete for 7 bit folders works after I changed to apply unescape only to 
the portion we escaped (i.e. folder name). 
But message delete still not working for 8 bit filders. It's failing in 
nsMessage::GetMsgDBHdr since the getting nsIMsgDBHdr is null.

nsMessage::GetMsgDBHdr(nsMessage * const 0x03be8cb4, nsIMsgDBHdr * * 0x0012f948) 
line 563
nsMsgLocalMailFolder::DeleteMessage(nsIMessage * 0x03be8cb4, nsIMsgWindow * 
0x00000000, int 1) line 2067 + 50 bytes
nsMsgLocalMailFolder::DeleteMessages(nsMsgLocalMailFolder * const 0x0319c90c, 
nsISupportsArray * 0x03be8d00, nsIMsgWindow * 0x00000000, int 1, int 1) line 
1780
DeleteMessage(nsIURI * 0x03b48764, nsIMsgFolder * 0x0319c90c) line 93 + 30 bytes
nsCopyMessageStreamListener::OnStopRequest(nsCopyMessageStreamListener * const 
0x03be4380, nsIChannel * 0x03b8c414, nsISupports * 0x03b48760, unsigned int 0, 
const unsigned short * 0x100a56b0 gCommonEmptyBuffer) line 186 + 26 bytes

Comment 72

[nhottanscp]

Attachment: additional patch to fix message delete problem

Comment 73

[nhottanscp]

Attachment: patch for review (fixed message delete problem)

Comment 74

[nhottanscp]

I attached a new patch (09/21/00 22:23) to fix the message delete problem.
I changed nsMailboxService.cpp. There were two problems in the old patch.
1) Place to unescape was not right, so the unescaped URI was stored in RDF and 
caused inconsistency.
2) The new patch only escapes folder name portion of URI which was encoded in 
AddSubFolder.

With this patch, I have no known problem for both 7 bit and 8 bit local mail 
folders.

Comment 75

[Alec Flett]

wow, this is great. the patches scare me a little of course, but they if they 
work, they work. r=alecf!

Comment 76

[Frank Tang]

Without this fix, non English user cannot use SeaMonkey to open their folder 
which is in non ASCII which they created by 4.x. Data lostage. 

Comment 77

[scottputterman]

if you've checked all of the things we talked about yesterday then it looks ok 
to me too. 

Comment 78

[nhottanscp]

I checked in to the trunk.
I tested my change on Windows, Macintosh, Linux.
I did not find problem with both 7 bit and 8 bit folder names.
I also tested some special 7 bit characters (' ', '[', ']') and they works fine 
(we did not escaped them before which is not good).

Comment 79

[nhottanscp]

More info about my change:
Before my change, we were sending raw 8 bit characters in native OS charset to 
RDF. The risk of sending raw 8 bit was that the data might be altered. And it 
actually happened (8 bit data corrupted after 8/23 caused this bug).
So my changes are,
* Escaped UTF-8 string to RDF
(by using escaped 7 bit data, we can avoid the corruption for 8 bit, reason of 
using UTF-8 is that RDF is more friendly for UTF-8 and it's easy to convert 
UTF-8 to UCS2).
* Unescape and convert to native OS charset to access local mailbox files
(this is needed to access the real mbox file, the native charset conversion was 
also used in the old code but it happened before sending data to RDF).

So the real new parts of my change are escape/unescape plus UTF-8 <->UCS2 
conversion which are used everywhere in our code. OS native charset conversion 
was just moved from one place to another.

The only change that might affect 7 bit data would be escape/unescape. I tested 
them (space, '[', ']') as I mentioned before).

Comment 80

[nhottanscp]

I tested  2000-09-22-21-M18 win32 build (it includes my change since I can use  
8 bit folders).
I tested for local mail, view message, send message, reply/forward message, 
delete message, copy message, move message, empty trash, rename folder, delete 
folder, print message, view source. With 7 bit folder names (including ' ', '[' 
and ']').
Everything worked except delete folder which does not work either with 
2000-09-22-13-M18 which was built before my check in. So, I filed a bug 53900.
I also tested IMAP and news briefly but did not see any problem.

Comment 81

[ji]

Unable to delete the non-ASCII folder on a POP account is an old problem. (Bug 
43349)

Comment 82

[nhottanscp]

I filed bug 53900 for 7 bit ascii folder name, bug 43349 is for 8 bit folder 
name.

Comment 83

[nhottanscp]

I tested Macintosh (did the same test as I did for windows).
I used 2000-09-22-20-M18 mac build (this includes my check in so it can access 8 
bit folders). I did not see any problem specific to this build. I found a 
printing bug and filed as bug 54021.

Comment 84

[ji]

I tested linux 2000092506 build. The original problem reported in this report 
has been fixed. No problem has been found with ASCII folders either.

Comment 85

[Peter Trudelle]

PDT agrees this fix needs to land on the branch - nsbeta3++

Comment 86

[scottputterman]

Naoki,

Is it possible that 54054 was caused by this checkin?  It looks like it just 
showed up on the trunk and only affects local folders.

Comment 87

[nhottanscp]

Yes, the unescape I put in my patch cause the problem, I should delay 
unescaping, I will update bug 54054.

Comment 88

[nhottanscp]

There was one problem in the old patch. Unescaping was applied to whole URI 
instead of actual folder name portion. This was not intended and caused a 
problem as bug 54054. I am going to attach a patch for this.

Comment 89

[nhottanscp]

Attachment: Fix to apply unescape only to a folder name portion.

Comment 90

[nhottanscp]

I checked in the latest path to both trunk and branch.

Comment 91

[lchiang]

ji - please verify when you get a chance. Thanks

Comment 92

[ji]

Verified with 09/29 win32, linux and Mac branch build. It is fixed.

Comment 93

[Asa Dotzler [:asa]]

reopening and resolving Fixed.  Bugs stay resolved until verified on both the
branch and the trunk.

Comment 94

[Asa Dotzler [:asa]]

Resolving as Fixed.  vtrunk keyword is in place requesting trunk verification
before this bug changes state to Verified.  When this is verified on the trunk
vtrunk keyword should be removed and the bug Status should be set to Verified.
Sorry for the spam, just trying to make sure we cover all of our bases here.

Comment 95

[ji]

Verified with win32, linux and Mac 2001-01-29-08-mtrunk build.