Closed Bug 52165 Opened 25 years ago Closed 25 years ago

Unable to access non-ASCII POP/Local mail folders

Categories

(MailNews Core :: Internationalization, defect, P2)

Product:
MailNews Core
Component:
Internationalization
Platform:
All
Windows 95
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: ji, Assigned: nhottanscp)

References

Details

(Keywords: regression, Whiteboard: [nsbeta3++] [pdtp2] patch for review)

Attachments

(13 files)

screen shot, two .msf files generated for one Japanese folder
26.33 KB, image/jpeg
Details
zip file contains a mail folder "náoki"
2.13 KB, application/octet-stream
Details
untested patch along the lines of what nhotta suggested
1.04 KB, patch
Details | Diff | Splinter Review
patch 1: JS changes to use GetUnicodeResource
24.85 KB, patch
Details | Diff | Splinter Review
patch 2: charset conversion in mailnews/base/util
1.14 KB, patch
Details | Diff | Splinter Review
patch 3: charset conversion in mailnews/local/src
3.17 KB, patch
Details | Diff | Splinter Review
the two call stacks I mentioned in my last comment
13.97 KB, text/plain
Details
patch 4: RDF change to add GetUnicodeValue
2.13 KB, patch
Details | Diff | Splinter Review
patch 5: JS change to use GetUnicodeValue, replacing patch 1
25.03 KB, patch
Details | Diff | Splinter Review
patch for review
5.45 KB, patch
Details | Diff | Splinter Review
additional patch to fix message delete problem
1.07 KB, patch
Details | Diff | Splinter Review
patch for review (fixed message delete problem)
5.20 KB, patch
Details | Diff | Splinter Review
Fix to apply unescape only to a folder name portion.
2.01 KB, patch
Details | Diff | Splinter Review
*******observed with linux and win32 2000091108 build***** Going to a Japanese POP or Local mail folder will get "unknown error". Steps to reproduce: 1. Launch Mail. 2. Select File | New | Folder... 3. Select a POP account or Local Folders as the parent folder 4. Create a folder with Japanese name 5. Select the new created Japanese folder You'll get a error msg window saying "unknown error" If you click on OK to close the error msg window and go to the folder again, you won't see the error msg, but the folder is not actually usable. (move/copying message to that folder won't work).
There is no this problem with latin-1 mail folders,
same thing happens with non-ascii name folders (french, german etc)
it is a weird problem: it looks like the message itself is copied because the total ammount of the messages in this folder is changing but when you open it it doesn't display anything.
On my Win95-J, the problem with latin-1 mail folders seems different with Ja folders. I don't get the "unknown error" when accessing the latin-1 folder and I can do move/copy msg between the latin-1 folders and the other folders. But after I restart the mail, the latin-1 mail folder name are not displayed well on the folder tree. The letters beginning from the accented character are stripped off, for example béx becomes b.
i guess there is a problem with Latin-1 folder on JA system for Local mail as there is a problem with JA folder on US system, when i attempt to copy/move a message to a JA folder on the POP account in my US system i'll crash (this is a known problem)and we are not currently fixing it, right Naoki?
Changed the platform to ALL.
Hardware: PC → All
Nominating for nsbeta3.
Keywords: nsbeta3
QA contact to ji. The problme is not observed with the 8/17/2000 Win32 build but is observed with the 8/23 Win32 build.
QA Contact: momoi → ji
I can reproduce this on my WinNT JA. Adding regression keyword. Adding kato san to cc, do you have time to look at this? I saw two .msf are generated, one with a correct Japanese name other is something garbage.
Status: NEW → ASSIGNED
Keywords: regression
P3 -> P2 I can reproduce this with Latin1 name using WinNT US. I will attach a test case.
Priority: P3 → P2
As I mentioned before there are two .msf files created. For a folder "náoki", "náoki.msg" and "náoki.msf" are generated. The second one is a file name encoded as UTF-8 which we do not support. We are using OS native charset for folder file name. Alec made a change to nsParseMailbox.cpp (rev 1.119) on 8/10. The change assumes folder file name to be UTF-8. But Kat Momoi mentioned the problem not is observed with 8/17 build so may not be directly related. Adding alecf to cc, do you know anything about this change? Do you know other place which assumes folder name to be UTF-8? I want to know the place which actually crates UTF-8 files.
This is a new bug. We don't have this problem in 4x, neither in PR2. This is not a JA specific problem. This bug also impact any 8-bit charset, such as western european . Mark it as nsbeta3+ P2.
Keywords: 4xp
Whiteboard: nsbeta3+
Per Naoki's comments, it can be reproduced with latin-1 mail folder on US NT. Changed the summary accordingly.
Summary: Unable to access Ja POP/Local mail folders → Unable to access non-ASCII POP/Local mail folders
>For a folder "náoki", "náoki.msg" and "náoki.msf" are generated. typo, should be both ".msf". For a folder "náoki", "náoki.msf" and "náoki.msf" are generated. I also forgot to cc alecf.
Xianglan found that the problem is not in 8/23 build but started with 8/24 build.
I checked Bonsai and found there were checkins for DOM string changes. I think RDF behavior was changed by this. Here is one of the changes. rdf/content/src/nsXULElement.cpp, rev=1.292 Before change: nsCAutoString tmp; tmp.AssignWithConversion(aName); After change: nsCAutoString tmp; tmp.Assign(NS_ConvertUCS2toUTF8(aName)); This might had affected local folder. I will investigate more tomorrow.
I think I found the place which sets UTF-8 path name. RDFServiceImpl::GetUnicodeResource creates UTF-8 path name. So nsMsgLocalMailFolder::Init is called in different ways (see two call stacks below). The second case, I understand that it's adding subfolders. But the first case I don't know what it is doing. Adding waterson to cc, do you have any idea about what it's doing for the first call stack? The break point is after I click a local mail folder. call stack 1: nsMsgLocalMailFolder::Init(nsMsgLocalMailFolder * const 0x04014460, const char * 0x0012b2a4) line 468 RDFServiceImpl::GetResource(RDFServiceImpl * const 0x00b6ace0, const char * 0x0012b2a4, nsIRDFResource * * 0x04014760) line 761 + 16 bytes RDFServiceImpl::GetUnicodeResource(RDFServiceImpl * const 0x00b6ace0, const unsigned short * 0x0012b324, nsIRDFResource * * 0x04014760) line 779 + 38 bytes nsXULElement::GetResource(nsXULElement * const 0x040147a4, nsIRDFResource * * 0x04014760) line 3526 + 34 bytes nsXULElement::ForceElementToOwnResource(nsXULElement * const 0x040147a0, int 0x00000001) line 1703 + 49 bytes nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x038c5b70, nsIContent * 0x03cf9640, nsIContent * 0x03fd01c0, int 0x00000001, nsIRDFResource * 0x03cfa100, int 0x00000000, Match * 0x03290eb8, nsIContent * * 0x0012caf0, int * 0x0012caf4) line 5418 nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x038c5c80, nsIContent * 0x03cf9640, nsIContent * 0x03cf9640, int 0x00000001, nsIRDFResource * 0x03cfa100, int 0x00000000, Match * 0x03290eb8, nsIContent * * 0x0012caf0, int * 0x0012caf4) line 5374 + 61 bytes nsXULTemplateBuilder::CreateContainerContents(nsIContent * 0x03cf9640, nsIRDFResource * 0x03cd56f0, int 0x00000000, nsIContent * * 0x0012caf0, int * 0x0012caf4) line 6145 nsXULTemplateBuilder::OpenContainer(nsXULTemplateBuilder * const 0x03b15924, nsIContent * 0x03cf9640) line 4317 + 54 bytes nsXULDocument::OpenWidgetItem(nsIContent * 0x03cf9640) line 4488 + 22 bytes nsXULDocument::AttributeChanged(nsXULDocument * const 0x03763070, nsIContent * 0x03cf9640, int 0x00000000, nsIAtom * 0x01a0edd0, int 0xffffffff) line 1645 nsXULElement::SetAttribute(nsXULElement * const 0x03cf9640, nsINodeInfo * 0x03fd0ba0, const basic_nsAReadableString<unsigned short> & {...}, int 0x00000001) line 2759 nsXULElement::SetAttribute(nsXULElement * const 0x03cf9644, const basic_nsAReadableString<unsigned short> & {...}, const basic_nsAReadableString<unsigned short> & {...}) line 1229 + 31 bytes ElementSetAttribute(JSContext * 0x03740a30, JSObject * 0x032958d8, unsigned int 0x00000002, long * 0x0323df3c, long * 0x0012d1d4) line 239 + 26 bytes js_Invoke(JSContext * 0x03740a30, unsigned int 0x00000002, unsigned int 0x00000000) line 731 + 23 bytes js_Interpret(JSContext * 0x03740a30, long * 0x0012db5c) line 2538 + 15 bytes js_Invoke(JSContext * 0x03740a30, unsigned int 0x00000001, unsigned int 0x00000002) line 748 + 13 bytes js_InternalInvoke(JSContext * 0x03740a30, JSObject * 0x032ba198, long 0x032ba720, unsigned int 0x00000000, unsigned int 0x00000001, long * 0x0012dcf0, long * 0x0012dc80) line 821 + 19 bytes JS_CallFunctionValue(JSContext * 0x03740a30, JSObject * 0x032ba198, long 0x032ba720, unsigned int 0x00000001, long * 0x0012dcf0, long * 0x0012dc80) line 3175 + 31 bytes nsJSContext::CallEventHandler(nsJSContext * const 0x03741130, void * 0x032ba198, void * 0x032ba720, unsigned int 0x00000001, void * 0x0012dcf0, int * 0x0012dcec, int 0x00000000) line 906 + 33 bytes nsJSEventListener::HandleEvent(nsIDOMEvent * 0x03cedcf4) line 154 + 64 bytes nsXBLEventHandler::ExecuteHandler(nsXBLEventHandler * const 0x03cfeef0, const basic_nsAReadableString<unsigned short> & {...}, nsIDOMEvent * 0x03cedcf4) line 593 nsXBLEventHandler::MouseClick(nsIDOMEvent * 0x03cedcf4) line 298 + 36 bytes nsEventListenerManager::HandleEvent(nsIPresContext * 0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, nsIDOMEventTarget * 0x03c390e8, unsigned int 0x00000002, nsEventStatus * 0x0012f784) line 861 + 23 bytes nsXULElement::HandleDOMEvent(nsXULElement * const 0x03c390e0, nsIPresContext * 0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 0x00000002, nsEventStatus * 0x0012f784) line 3307 nsXULElement::HandleDOMEvent(nsXULElement * const 0x03cf9640, nsIPresContext * 0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 0x00000002, nsEventStatus * 0x0012f784) line 3324 + 39 bytes nsXULElement::HandleDOMEvent(nsXULElement * const 0x03a8bdb0, nsIPresContext * 0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 0x00000002, nsEventStatus * 0x0012f784) line 3324 + 39 bytes nsXULElement::HandleDOMEvent(nsXULElement * const 0x03fa4d10, nsIPresContext * 0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 0x00000002, nsEventStatus * 0x0012f784) line 3324 + 39 bytes nsXULElement::HandleDOMEvent(nsXULElement * const 0x03fa52f0, nsIPresContext * 0x03765680, nsEvent * 0x0012f48c, nsIDOMEvent * * 0x0012f3b0, unsigned int 0x00000001, nsEventStatus * 0x0012f784) line 3324 + 39 bytes PresShell::HandleEventInternal(nsEvent * 0x0012f48c, nsIView * 0x00000000, nsEventStatus * 0x0012f784) line 4182 + 45 bytes PresShell::HandleEventWithTarget(PresShell * const 0x03766ab0, nsEvent * 0x0012f48c, nsIFrame * 0x03279130, nsIContent * 0x03fa52f0, nsEventStatus * 0x0012f784) line 4163 + 18 bytes nsEventStateManager::CheckForAndDispatchClick(nsEventStateManager * const 0x03b94080, nsIPresContext * 0x03765680, nsMouseEvent * 0x0012f894, nsEventStatus * 0x0012f784) line 1820 + 59 bytes nsEventStateManager::PostHandleEvent(nsEventStateManager * const 0x03b94088, nsIPresContext * 0x03765680, nsEvent * 0x0012f894, nsIFrame * 0x03279130, nsEventStatus * 0x0012f784, nsIView * 0x03d0b330) line 901 + 28 bytes PresShell::HandleEventInternal(nsEvent * 0x0012f894, nsIView * 0x03d0b330, nsEventStatus * 0x0012f784) line 4202 + 43 bytes PresShell::HandleEvent(PresShell * const 0x03766ab4, nsIView * 0x03d0b330, nsGUIEvent * 0x0012f894, nsEventStatus * 0x0012f784, int 0x00000001, int & 0x00000001) line 4117 + 23 bytes nsView::HandleEvent(nsView * const 0x03d0b330, nsGUIEvent * 0x0012f894, unsigned int 0x0000001c, nsEventStatus * 0x0012f784, int 0x00000001, int & 0x00000001) line 379 nsViewManager2::DispatchEvent(nsViewManager2 * const 0x03765330, nsGUIEvent * 0x0012f894, nsEventStatus * 0x0012f784) line 1429 HandleEvent(nsGUIEvent * 0x0012f894) line 68 nsWindow::DispatchEvent(nsWindow * const 0x03d0bb14, nsGUIEvent * 0x0012f894, nsEventStatus & nsEventStatus_eIgnore) line 614 + 10 bytes nsWindow::DispatchWindowEvent(nsGUIEvent * 0x0012f894) line 635 nsWindow::DispatchMouseEvent(unsigned int 0x0000012d, nsPoint * 0x00000000) line 3811 + 21 bytes ChildWindow::DispatchMouseEvent(unsigned int 0x0000012d, nsPoint * 0x00000000) line 4021 nsWindow::ProcessMessage(unsigned int 0x00000202, unsigned int 0x00000000, long 0x001b0007, long * 0x0012fc10) line 2889 + 24 bytes nsWindow::WindowProc(HWND__ * 0x006d07b4, unsigned int 0x00000202, unsigned int 0x00000000, long 0x001b0007) line 883 + 27 bytes call stack 2: nsMsgLocalMailFolder::Init(nsMsgLocalMailFolder * const 0x03cfa100, const char * 0x0012c118) line 468 RDFServiceImpl::GetResource(RDFServiceImpl * const 0x00b6ace0, const char * 0x0012c118, nsIRDFResource * * 0x0012c0f0) line 761 + 16 bytes nsMsgLocalMailFolder::AddSubfolder(nsMsgLocalMailFolder * const 0x03cd570c, nsAutoString * 0x0012c1bc, nsIMsgFolder * * 0x0012c1b4) line 530 + 69 bytes nsMsgLocalMailFolder::CreateSubFolders(nsFileSpec & {...}) line 490 + 62 bytes nsMsgLocalMailFolder::GetSubFolders(nsMsgLocalMailFolder * const 0x03cd570c, nsIEnumerator * * 0x0012c328) line 705 + 15 bytes nsMsgFolderDataSource::createFolderChildNode(nsIMsgFolder * 0x03cd570c, nsIRDFNode * * 0x0012c3e4) line 1468 + 36 bytes nsMsgFolderDataSource::createFolderNode(nsIMsgFolder * 0x03cd570c, nsIRDFResource * 0x0229d390, nsIRDFNode * * 0x0012c3e4) line 900 + 16 bytes nsMsgFolderDataSource::GetTarget(nsMsgFolderDataSource * const 0x03cce280, nsIRDFResource * 0x03cd56f0, nsIRDFResource * 0x0229d390, int 0x00000001, nsIRDFNode * * 0x0012c3e4) line 305 + 25 bytes CompositeDataSourceImpl::GetTarget(CompositeDataSourceImpl * const 0x03b15150, nsIRDFResource * 0x03cd56f0, nsIRDFResource * 0x0229d390, int 0x00000001, nsIRDFNode * * 0x0012c3e4) line 721 + 28 bytes nsXULTemplateBuilder::CheckContainer(nsIRDFResource * 0x03cd56f0, int * 0x0012cb20, int * 0x0012cb24) line 6284 + 65 bytes nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x038c5b70, nsIContent * 0x03888fa0, nsIContent * 0x03c390e0, int 0x00000001, nsIRDFResource * 0x03cd56f0, int 0x00000000, Match * 0x03290b00, nsIContent * * 0x0012dc18, int * 0x0012dc1c) line 5423 + 29 bytes nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x038c5c80, nsIContent * 0x03888fa0, nsIContent * 0x03888fa0, int 0x00000001, nsIRDFResource * 0x03cd56f0, int 0x00000000, Match * 0x03290b00, nsIContent * * 0x0012dc18, int * 0x0012dc1c) line 5374 + 61 bytes nsXULTemplateBuilder::CreateContainerContents(nsIContent * 0x03888fa0, nsIRDFResource * 0x03ccebb0, int 0x00000000, nsIContent * * 0x0012dc18, int * 0x0012dc1c) line 6145 nsXULTemplateBuilder::CreateTemplateAndContainerContents(nsIContent * 0x03888fa0, nsIContent * * 0x0012dc18, int * 0x0012dc1c) line 6038 + 34 bytes nsXULTemplateBuilder::RebuildContainerInternal(nsIContent * 0x03888fa0, int 0x00000000) line 4476 + 40 bytes nsXULTemplateBuilder::RebuildContainer(nsXULTemplateBuilder * const 0x03b15924, nsIContent * 0x03888fa0) line 4433 + 17 bytes nsXULDocument::RebuildWidgetItem(nsIContent * 0x03888fa0) line 4591 + 22 bytes nsXULDocument::AttributeChanged(nsXULDocument * const 0x03763070, nsIContent * 0x03888fa0, int 0x00000000, nsIAtom * 0x025cad50, int 0xffffffff) line 1671 nsXULElement::SetAttribute(nsXULElement * const 0x03888fa0, nsINodeInfo * 0x03b53260, const basic_nsAReadableString<unsigned short> & {...}, int 0x00000001) line 2759 nsXULElement::SetAttribute(nsXULElement * const 0x03888fa4, const basic_nsAReadableString<unsigned short> & {...}, const basic_nsAReadableString<unsigned short> & {...}) line 1229 + 31 bytes nsXULTreeElement::SetAttribute(nsXULTreeElement * const 0x03c9ca28, const basic_nsAReadableString<unsigned short> & {...}, const basic_nsAReadableString<unsigned short> & {...}) line 54 + 29 bytes ElementSetAttribute(JSContext * 0x03740a30, JSObject * 0x00f31f60, unsigned int 0x00000002, long * 0x03279174, long * 0x0012e344) line 239 + 26 bytes js_Invoke(JSContext * 0x03740a30, unsigned int 0x00000002, unsigned int 0x00000000) line 731 + 23 bytes js_Interpret(JSContext * 0x03740a30, long * 0x0012eccc) line 2538 + 15 bytes js_Invoke(JSContext * 0x03740a30, unsigned int 0x00000001, unsigned int 0x00000002) line 748 + 13 bytes js_InternalInvoke(JSContext * 0x03740a30, JSObject * 0x00f55628, long 0x00d202a8, unsigned int 0x00000000, unsigned int 0x00000001, long * 0x0012ee60, long * 0x0012edf0) line 821 + 19 bytes JS_CallFunctionValue(JSContext * 0x03740a30, JSObject * 0x00f55628, long 0x00d202a8, unsigned int 0x00000001, long * 0x0012ee60, long * 0x0012edf0) line 3175 + 31 bytes nsJSContext::CallEventHandler(nsJSContext * const 0x03741130, void * 0x00f55628, void * 0x00d202a8, unsigned int 0x00000001, void * 0x0012ee60, int * 0x0012ee5c, int 0x00000000) line 906 + 33 bytes nsJSEventListener::HandleEvent(nsIDOMEvent * 0x03c5b054) line 154 + 64 bytes nsEventListenerManager::HandleEventSubType(nsListenerStruct * 0x03773350, nsIDOMEvent * 0x03c5b054, nsIDOMEventTarget * 0x037411b0, unsigned int 0x00000001, unsigned int 0x00000007) line 788 + 19 bytes nsEventListenerManager::HandleEvent(nsIPresContext * 0x03765680, nsEvent * 0x0012f3b0, nsIDOMEvent * * 0x0012f374, nsIDOMEventTarget * 0x037411b0, unsigned int 0x00000007, nsEventStatus * 0x0012f3d4) line 1361 + 39 bytes GlobalWindowImpl::HandleDOMEvent(GlobalWindowImpl * const 0x037411a0, nsIPresContext * 0x03765680, nsEvent * 0x0012f3b0, nsIDOMEvent * * 0x0012f374, unsigned int 0x00000001, nsEventStatus * 0x0012f3d4) line 467 DocumentViewerImpl::LoadComplete(DocumentViewerImpl * const 0x03764c00, unsigned int 0x00000000) line 668 + 47 bytes nsWebShell::OnEndDocumentLoad(nsWebShell * const 0x037418d8, nsIDocumentLoader * 0x037416b0, nsIChannel * 0x02e3ae10, unsigned int 0x00000000) line 981 nsDocLoaderImpl::FireOnEndDocumentLoad(nsDocLoaderImpl * 0x037416b0, nsIChannel * 0x02e3ae10, unsigned int 0x00000000) line 804 nsDocLoaderImpl::DocLoaderIsEmpty(unsigned int 0x00000000) line 611 nsDocLoaderImpl::OnStopRequest(nsDocLoaderImpl * const 0x037416b4, nsIChannel * 0x03c5f7c0, nsISupports * 0x00000000, unsigned int 0x00000000, const unsigned short * 0x100a56b0 gCommonEmptyBuffer) line 554 nsLoadGroup::RemoveChannel(nsLoadGroup * const 0x03741650, nsIChannel * 0x03c5f7c0, nsISupports * 0x00000000, unsigned int 0x00000000, const unsigned short * 0x100a56b0 gCommonEmptyBuffer) line 566 + 39 bytes nsFileChannel::OnStopRequest(nsFileChannel * const 0x03c5f7c8, nsIChannel * 0x03c5f4d0, nsISupports * 0x00000000, unsigned int 0x00000000, const unsigned short * 0x100a56b0 gCommonEmptyBuffer) line 656 nsOnStopRequestEvent::HandleEvent(nsOnStopRequestEvent * const 0x03ca4030) line 302 nsStreamListenerEvent::HandlePLEvent(PLEvent * 0x03ca50e0) line 97 + 12 bytes PL_HandleEvent(PLEvent * 0x03ca50e0) line 575 + 10 bytes PL_ProcessPendingEvents(PLEventQueue * 0x00b46840) line 508 + 9 bytes _md_EventReceiverProc(HWND__ * 0x00440676, unsigned int 0x0000c102, unsigned int 0x00000000, long 0x00b46840) line 1044 + 9 bytes USER32! 77e71820()
Whiteboard: nsbeta3+ → nsbeta3+ Need help (not familiar with RDF)
I think RDF to use UTF-8 URI is the right thing. So my current plan is to change nsMsgLocalMailFolder to pass UTF-8 URI to RDF instead of OS native charset. Then convert the UTF-8 string to OS native charset for the DB construction. 1) nsMsgLocalMailFolder::AddSubfolder, Change to call GetResource() with UTF-8 URI. 2) nsMsgLocalMailFolder::Init, Add a conversion from UTF-8 to OS native charset before calling nsMsgDBFolder::Init. I made those changes to my local tree. But nsMsgLocalMailFolder::Init is still called with OS native charset in few cases. I need to investigate those cases and change to use UTF-8.
ugh - this basically means any place where we call "GetResource" where the folder hasn't been created yet. this might come from preferences like the Send/Drafts/etc folders
>2) nsMsgLocalMailFolder::Init, This was wrong. I should change nsMsgFolder::Init() instead. It calls nsRDFResource::Init() and CreateBaseMessageURI(). So I passed UTF-8 to nsRDFResource::Init() and OS charset to CreateBaseMessageURI(). I did above change. Now I am getting back double-converted UTF-8 from RDF. For 'á' I am getting 0xC383C2A1 instead of 0xC3A1. This usually happens when using AssignWithConversion() to convert 8 bit data to unicode then use NS_ConvertUCS2toUTF8 to create UTF-8 string. I feel that RDF is not always expecting UTF-8 URI. A question to waterson: Is nsRDFResource::Init() expecting a URI as UTF-8? If not, is there other interface with takes UTF-8 URI. Part of the call stack which passing a corrupted UTF-8 URI: nsMsgLocalMailFolder::Init(nsMsgLocalMailFolder * const 0x03d6e9d0, const char * 0x0012b2a4) line 479 RDFServiceImpl::GetResource(RDFServiceImpl * const 0x00b6a4a0, const char * 0x0012b2a4, nsIRDFResource * * 0x03d6ec90) line 761 + 16 bytes RDFServiceImpl::GetUnicodeResource(RDFServiceImpl * const 0x00b6a4a0, const unsigned short * 0x0012b324, nsIRDFResource * * 0x03d6ec90) line 779 + 38 bytes nsXULElement::GetResource(nsXULElement * const 0x03d6ecd4, nsIRDFResource * * 0x03d6ec90) line 3526 + 34 bytes nsXULElement::ForceElementToOwnResource(nsXULElement * const 0x03d6ecd0, int 0x00000001) line 1703 + 49 bytes nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x03986b30, nsIContent * 0x03d5aa20, nsIContent * 0x03d6fbb0, int 0x00000001, nsIRDFResource * 0x03d5ba30, int 0x00000000, Match * 0x03567580, nsIContent * * 0x0012caf0, int * 0x0012caf4) line 5418 nsXULTemplateBuilder::BuildContentFromTemplate(nsIContent * 0x03986c40, nsIContent * 0x03d5aa20, nsIContent * 0x03d5aa20, int 0x00000001, nsIRDFResource * 0x03d5ba30, int 0x00000000, Match * 0x03567580, nsIContent * * 0x0012caf0, int * 0x0012caf4) line 5374 + 61 bytes
> A question to waterson: Is nsRDFResource::Init() expecting a URI as UTF-8? Yes. It will store the bytes you give to it.
Yes!! I am seeing this double conversion in the filters as well. Something is screwy in RDF... Also, I'm concerned about you changing CreateBaseMessageURI to use the platform charset - there is other code (namely nsMessage::GetMsgFolder()) which assumes that the base message URI is a direct copy of the folder URI, with some modification. The message and folder URIs should be in the same charset no matter what. I have a feeling that this double-conversion of UTF8 is the real source of the problem, as this USED to work.
The only conversion that core RDF does (modulo parsing RDF/XML) is in nsRDFService::GetUnicodeResource(). In this case, RDF is given a UCS-2 string, and it uses NS_ConvertUCS2toUTF8() to deflate it to a single-byte representation. nsRDFResource::Init() accepts a single-byte string, and is agnostic as to its representation. If you've created the resource "properly" (i.e., using nsRDFService), then it will contain a UTF-8 encoded string. If you create the RDF resource using your own mechanism, it's up to you to ensure that you play nice with the bytes.
>Also, I'm concerned about you changing CreateBaseMessageURI to use the platform >charset This is not a change, it's been using the platform charset. My change is to pass UTF-8 to nsRDFResource::Init().
>If you've created the resource "properly" So does this mean creating RDF without using nsRDFResource::Init()? As I am not familiar with RDF (also local mailfodler code), I dont' know what kind of change requred. Would it affects a lot to local mailfolder code?
The "normal" way of getting at nsRDFResource objects is to call nsIRDFService::Get[Unicode]Resource(). *This* code does the work of creating an nsRDFResource object, and then calling its Init() method with a string. AFAIK, this is the *only* was that nsRDFResource objects get created, but I may be wrong wrt mailnews. alecf? Now, I mis-spoke a bit, above. If you call nsIRDFService::GetResource(), you'll be handing in a "naked" single-byte string, and RDF won't know how it's encoded. So it'll just take the bytes and pass them along.
right, mailnews doesn't do anything special to create resources, we always just call GetResource But my point is that CreateBaseMessageURI should use exactly the same charset as the folder. whatever nsMsgFolder::Init() gets, it should pass this on to CreateBaseMessageURI - there should be no conversion.
I agree, the URI should also contain UTF-8 string. The problem is that the URI is tightly connected with actual file name. Since we cannot use UTF-8 as a real file name, we need to find a place where the URI is used to get a file name then apply the conversion there. But above issue is not related to the double-conversion problem. In the partial call stack I put today, nsXULElement::GetResource calling GetUnicodeResource so nsMsgLocalMailFolder::Init is getting double-converted UTF-8 URI. nsMsgLocalMailFolder cannot control GetResource or GetUnicodeResource is used.
ooh! This totally explains what's going on! Oh man. This is not a mail problem, this is a XUL template or nsXULElement problem. Here's what's happening: the template builder is probably saving the direct ASCII version of the URI of the resource in the nsXULElement that it creates. This gets converted to unicode (i.e. a direct conversion, no charsets involved) when the attribute is read. Then, the template builder is calling GetUnicodeResource which means the UTF8 string, which is currently a UTF8 string stored in PRUnichar format, gets re-encoded in UTF8 by GetUnicodeResource. This is really bad. I think that either GetUnicodeResource has to do an ASCII conversion instead of UTF8, for nsbeta3. After nsbeta3 we need to fix the template/XUL code to do the right conversions in the right places. this also explains bug 52190, because the filter datasource is getting queried on a string that has been double-converted into UTF8
With respect to nsIRDFService::GetUnicodeResource(), we can't change it, because that would undo several nsbeta2 bug fixes. Sorry! :-( I also think that nsXULElement::GetResource() is implemented correctly. That said, I could believe that somebody isn't properly *encoding* to UCS-2 before setting the attribute's value...I gotta run, but maybe poke around in nsXULContentUtils.cpp (which is what nsXULTemplateBuilder relies on to do a lot of the URI-to-attribute value transformation).
ugh! ok. not surprised. I'll poke around at RDF and see what I can find adding robert, the other template wizard. robert - read my previous comment or two
I found one place in RDF which applies AssignWithConversion to uri. This creates a nsString with a data actually contains UTF-8. Later if we do NS_ConvertUCS2toUTF8 then we get the double-converted string. nsXULTemplateBuilder.cpp 5155 nsXULTemplateBuilder::BuildContentFromTemplate 5410 nsAutoString id; id.AssignWithConversion(uri); There are two other places which are related to this bug does AssignWithConversion. They may be okay, as long as we convert the string to OS charset before doing it. nsMsgFolder.cpp 525 nsMsgFolder::parseURI(PRBool needServer) 583 mName.AssignWithConversion(result); nsLocalFolderSummarySpec.cpp 65 void nsLocalFolderSummarySpec:: CreateSummaryFileName() 71 nsString fullLeafName; fullLeafName.AssignWithConversion(leafName);
the RDF one is exactly it! it's the ID that we're concerned about, so this is probably it. Anyway, try converting that one to convert from UTF8 to UCS2, and see if that makes the problem go away. the assignment to mName may also have something to do with it, we should probably be converting from UTF8 there too. I have no idea about the third case, but I would guess that we should fix it there too. I think the RDF one is the most important though. if that fixes the problem, I say we should just fix that.
After I changed nsXULTemplateBuilder::BuildContentFromTemplate to use NS_ConvertUTF8toUCS2 instead of AssignWithConversion, looks like the double conversion problem has gone. But looks like I need to take care other things too. I am getting UTF-8 string for CreateSummaryFileName and a UTF-8 .msf is created.
Yep, nsXULTemplateBuilder.cpp:4949 looks suspect also...
Naoki, can you attach a patch here, and we can all try it out? Chris's other finding would be good too.
I applied the patch but still does not work. This bug was introduced on 8/23. I think that's because RDF started to return UTF-8 URI occasionally which I think related to the DOM string change. So the bug cannot be fixed as long as local mail depends on OS charset.
Local mail needs to use file names as OS charset even URI is UTF-8. So I need help from mailnew eng. to identify the place(s) to put a conversion between UTF-8 and OS charset.
Naoki and I did some debugging and this is what I'm seeing: We are converting the uri to UTF8 for each folder. When we call NotifyFolderEvent we are passing in this uri. However, before we can do anything like change folders we first check to see if the uri passed into NotifyFolderEvent is the same as the uri we saved off(we save off the uri to make sure we don't do anything for a folder that isn't loaded into the current window). We get the saved off uri from getAttribute on the selected folder tree item. We do this by doing uri = selectedFolder.getAttribute("id"). This does not appear to be coming back in UTF8 and hence our comparisons when the folder is loaded, as mentioned above, fails. So, my question is, should we be using something other than getAttribute to get back the UTF8 uri we stick into the resource?
ok, I just talked to Naoki. It seems that what I wrote was wrong. GetAttribute is returning the correct value but we are not sending the correct value to NotifyFolderEvent. We are looking in to that. Unless we change our minds again, ignore my previous comments (except for that fact that the 2 strings aren't equal which seems to be the problem).
pdt agrees p2
Whiteboard: nsbeta3+ Need help (not familiar with RDF) → nsbeta3+ Need help (not familiar with RDF)[pdtp2]
It was hard to check in the console which one was the correct UTF-8 string. But it turns out that the Scott's original comment is right. Sorry about the confusion. We are sending a correct UTF-8 to NotifyFolderEvent (0xC3A1 for á). But from getAttribute() we get non UTF-8 value (0xE1 for á).
I just mentioned this to scott - you could also try accessing the "id" attribute as in folder.id instead of folder.getAttribute("id"); - it goes through a slightly different path in the DOM code.
I tried folder.id to get URI but the same result. It did not return UTF-8 URI.
So the current issue is that there are two URI in JS code which do not match. The uri passed into NotifyFolderEvent is UTF-8. And, uri = selectedFolder.getAttribute("id"), this is UCS2. In JS file, I think URI should not be raw UTF-8. Is there way to get the URI in UCS2 in JS?
if it's not doing that, it's a bug. the "id" attribute is supposed to return the UCS2 representation of the attribute, from JS. if it's not it's a bug where someone is using AssignWithConversion (or one of the other AppendWithConversion/etc) operators incorrectly.
selectedFolder.getAttribute is returning UCS2. The other one returns UTF-8 that's why the two do not match.
Currently, I am blocked by the above issue (URI in UCS2 and UTF8 not match), any help would be appreciated. In the JS file, I hacked the JS code to proceed. Then I can see messages in thread pane. msgMail3PaneWindow.js uri = gCurrentFolderToReroot; if(uri == gCurrentFolderToReroot) Also changed nsMailboxService::FetchMessage which is getting a non UTF-8 URI. So I put a conversion from OS charset to UTF-8 there (not sure who is passing OS charset and if this is local mail specific or not). Then I can see a message, finally.
can you attach a patch with all of the relevant changes?
what you are saying is that we need nsRDFResource::Value to be in UCS2? Right now it's just returning what we've stored in it, which is UTF8.
Yes, that UTF-8 URI doesn't match with the URI by getAttribute.
I attached the changes in my local tree so far. I also has the waterson's patch previously attached. The bug is reproducible on US system with the previously attached mailbox data.
I applied the patches so far and tested on Japanese system. I tried a Japanese local mail folder. The thread pane displays messages fine. But when I clicked a message, I got two asserts then the "Unknown error" dialog shows up. I will attach the call stack. The first one is inXPCConvert::JSData2Native, xpcconvert.cpp. NS_ASSERTION(legalRange,"U+0080/U+0100 - U+FFFF data lost"); The string is a valid unicode (japanese hiragana). But failing because of the following check. //#define STRICT_CHECK_OF_UNICODE #ifdef STRICT_CHECK_OF_UNICODE #define ILLEGAL_RANGE(c) (0!=((c) & 0xFF80)) #else // STRICT_CHECK_OF_UNICODE #define ILLEGAL_RANGE(c) (0!=((c) & 0xFF00)) #endif // STRICT_CHECK_OF_UNICODE The second one is in nsMailboxProtocol::SetupMessageExtraction, in nsMailboxProtocol.cpp. rv = m_runningUrl->GetMessageHeader(getter_AddRefs(msgHdr)); This failed, I think due to an invalid URL.
In xpcconvert.cpp, by defining STRICT_CHECK_OF_UNICODE, I can also activate the assersion for NativeData2JS. I hit the assersion using the Japanese folder name. It's passing the japanese folder name in UTF-8. But this routine does not expect UTF-8 as an input. The JS caller was nsMsgMail3PaneWindow.js line 102. 102 var uri = resource.Value; That's the same place we got the problem of not matching UTF-8 and UCS2 URI.
I added GetUnicodeValue() to nsIRDFResource and used it in the JS file. That seems to solve the URI encoding inconsistency problem. I have no clue about the Japanese folder name problem. I will apply the nsIRDFResource change to my Japanese build and see if that can change anything.
it sounds like this is the right thing - otherwise there's no way to get the UCS2 version of the string from an RDF Resource. (as a side note, I believe you can say *aURI = NS_ConvertUTF8toUCS2(mURI).ToNewUnicode() to avoid an extra strcpy)
For Japanese URI problem, the URI is UCS2 but nsIMessenger::OpenURL(in string url), this is a char* interface. So all the Japanese characters were stripped out (e.g. 0x3042 => 42). Latin1 is fine since upper byte is zero (e.g. 00E1 => E1). So either need to add unicode version of OpenURL or escape the folder name part.
I changed to escape the UTF-8 string in AddSubfolder before passing to RDF. This works fine since our code works when the data is 7 bit (that's why IMAP is working fine with japanese folder name by using UTF-7), thanks to Frank for the suggestion. So we do not need to change any JS file and RDF. Just need escape/unesape and charset conversions (native charset and UTF-8). I will attach a diff for review.
Attached patch patch for review — — Splinter Review
Whiteboard: nsbeta3+ Need help (not familiar with RDF)[pdtp2] → nsbeta3+ [pdtp2] patch for review
good job tracking this stuff down naoki. I'll try to finish reviewing these thursday morning so you can checkin.
Message delete does not work with the patch. I am investigating.
Message delete for 7 bit folders works after I changed to apply unescape only to the portion we escaped (i.e. folder name). But message delete still not working for 8 bit filders. It's failing in nsMessage::GetMsgDBHdr since the getting nsIMsgDBHdr is null. nsMessage::GetMsgDBHdr(nsMessage * const 0x03be8cb4, nsIMsgDBHdr * * 0x0012f948) line 563 nsMsgLocalMailFolder::DeleteMessage(nsIMessage * 0x03be8cb4, nsIMsgWindow * 0x00000000, int 1) line 2067 + 50 bytes nsMsgLocalMailFolder::DeleteMessages(nsMsgLocalMailFolder * const 0x0319c90c, nsISupportsArray * 0x03be8d00, nsIMsgWindow * 0x00000000, int 1, int 1) line 1780 DeleteMessage(nsIURI * 0x03b48764, nsIMsgFolder * 0x0319c90c) line 93 + 30 bytes nsCopyMessageStreamListener::OnStopRequest(nsCopyMessageStreamListener * const 0x03be4380, nsIChannel * 0x03b8c414, nsISupports * 0x03b48760, unsigned int 0, const unsigned short * 0x100a56b0 gCommonEmptyBuffer) line 186 + 26 bytes
I attached a new patch (09/21/00 22:23) to fix the message delete problem. I changed nsMailboxService.cpp. There were two problems in the old patch. 1) Place to unescape was not right, so the unescaped URI was stored in RDF and caused inconsistency. 2) The new patch only escapes folder name portion of URI which was encoded in AddSubFolder. With this patch, I have no known problem for both 7 bit and 8 bit local mail folders.
wow, this is great. the patches scare me a little of course, but they if they work, they work. r=alecf!
Without this fix, non English user cannot use SeaMonkey to open their folder which is in non ASCII which they created by 4.x. Data lostage.
if you've checked all of the things we talked about yesterday then it looks ok to me too.
I checked in to the trunk. I tested my change on Windows, Macintosh, Linux. I did not find problem with both 7 bit and 8 bit folder names. I also tested some special 7 bit characters (' ', '[', ']') and they works fine (we did not escaped them before which is not good).
More info about my change: Before my change, we were sending raw 8 bit characters in native OS charset to RDF. The risk of sending raw 8 bit was that the data might be altered. And it actually happened (8 bit data corrupted after 8/23 caused this bug). So my changes are, * Escaped UTF-8 string to RDF (by using escaped 7 bit data, we can avoid the corruption for 8 bit, reason of using UTF-8 is that RDF is more friendly for UTF-8 and it's easy to convert UTF-8 to UCS2). * Unescape and convert to native OS charset to access local mailbox files (this is needed to access the real mbox file, the native charset conversion was also used in the old code but it happened before sending data to RDF). So the real new parts of my change are escape/unescape plus UTF-8 <->UCS2 conversion which are used everywhere in our code. OS native charset conversion was just moved from one place to another. The only change that might affect 7 bit data would be escape/unescape. I tested them (space, '[', ']') as I mentioned before).
I tested 2000-09-22-21-M18 win32 build (it includes my change since I can use 8 bit folders). I tested for local mail, view message, send message, reply/forward message, delete message, copy message, move message, empty trash, rename folder, delete folder, print message, view source. With 7 bit folder names (including ' ', '[' and ']'). Everything worked except delete folder which does not work either with 2000-09-22-13-M18 which was built before my check in. So, I filed a bug 53900. I also tested IMAP and news briefly but did not see any problem.
Unable to delete the non-ASCII folder on a POP account is an old problem. (Bug 43349)
I filed bug 53900 for 7 bit ascii folder name, bug 43349 is for 8 bit folder name.
I tested Macintosh (did the same test as I did for windows). I used 2000-09-22-20-M18 mac build (this includes my check in so it can access 8 bit folders). I did not see any problem specific to this build. I found a printing bug and filed as bug 54021.
I tested linux 2000092506 build. The original problem reported in this report has been fixed. No problem has been found with ASCII folders either.
PDT agrees this fix needs to land on the branch - nsbeta3++
Whiteboard: nsbeta3+ [pdtp2] patch for review → [nsbeta3++] [pdtp2] patch for review
Naoki, Is it possible that 54054 was caused by this checkin? It looks like it just showed up on the trunk and only affects local folders.
Yes, the unescape I put in my patch cause the problem, I should delay unescaping, I will update bug 54054.
There was one problem in the old patch. Unescaping was applied to whole URI instead of actual folder name portion. This was not intended and caused a problem as bug 54054. I am going to attach a patch for this.
Blocks: 54054
I checked in the latest path to both trunk and branch.
Status: ASSIGNED → RESOLVED
Closed: 25 years ago
Resolution: --- → FIXED
ji - please verify when you get a chance. Thanks
Verified with 09/29 win32, linux and Mac branch build. It is fixed.
Status: RESOLVED → VERIFIED
Keywords: vtrunk
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
reopening and resolving Fixed. Bugs stay resolved until verified on both the branch and the trunk.
Resolving as Fixed. vtrunk keyword is in place requesting trunk verification before this bug changes state to Verified. When this is verified on the trunk vtrunk keyword should be removed and the bug Status should be set to Verified. Sorry for the spam, just trying to make sure we cover all of our bases here.
Status: REOPENED → RESOLVED
Closed: 25 years ago → 25 years ago
Resolution: --- → FIXED
Verified with win32, linux and Mac 2001-01-29-08-mtrunk build.
Status: RESOLVED → VERIFIED
Keywords: vtrunk
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: