Closed Bug 117385 Opened 23 years ago Closed 22 years ago

Can't delete/rename a folder with name containing 0x5b/0x5d/0x7c

Categories

(MailNews Core :: Backend, defect, P2)

x86
Windows XP
defect

Tracking

(Not tracked)

VERIFIED FIXED
mozilla1.0

People

(Reporter: kazhik, Assigned: naving)

References

(Blocks 1 open bug)

Details

(Keywords: intl, Whiteboard: [ish1+][verifiedish1])

Attachments

(1 file, 5 obsolete files)

Can't delete a folder with Japanese name. Steps to reproduce: (1) Create a folder with Japanese name. (2) Delete the folder. It's moved to trash box. (3) Restart Mozilla. Actual result: The folder is moved from trash box to its original position. Expected result: The folder stays in trash box. Build: 2001122803/WinXP
Is this IMAP or local folder (or both)?
Local folder for pop account.
KOIKE san, is the test environment WinXP English or Japanese? Local fFolder name support depends on System charset.
Status: NEW → ASSIGNED
I don't see this problem with 01/07 build on a US W2K with default locale set to Japanese.
This bug happens with the character whose second second byte is 0x5b(SJIS) or 0xfc(UCS2). For example "folder" or "center" in Japanese. Please see the original report in Bugzilla-jp: http://bugzilla.mozilla.gr.jp/show_bug.cgi?id=1729
Yes, I can reproduce the problem with the folder name as "メール" (view this page in Shift-JIS). After File | Empty Trash, the folder remains, but the mails in the folder are deleted.
Nominating for nsbeta1.
Keywords: nsbeta1
There is a code which checks for '[' 0x5B and other characters. http://lxr.mozilla.org/seamonkey/source/mailnews/base/util/nsMsgUtils.cpp#230 I am not sure if this is something to do with this bug, cc to sspitzer. BTW, is this a delete only problem? Other operations work fine?
Rename is not working properly for this kind of mail folders either. After rename and restart, both new name and old name show up on the folder tree, but selecting the folders doesn't refresh the thread pane. The file with the old name still exist on the system.
Summary: Can't delete a folder with Japanese name → Can't delete/rename a folder with Japanese name containing a char with 2nd byte as 0x5b
Xianglan, could you also try Shift_JIS characters which contain ']' 0x5D (e.g. 0x835D) and '|' 0x7C (e.g. 0x837C)? Those characters are also checked in the code I mentioned.
Same problems with the characters that have the 2nd byte as ']' 0x5D or '|' 0x7C.
Summary: Can't delete/rename a folder with Japanese name containing a char with 2nd byte as 0x5b → Can't delete/rename a folder with Japanese name containing a char with 2nd byte as 0x5b/0x5d/0x7c
I can reproduce this with ASCII folder name like "a[b]c", I used NS6.2. Reassign to putterman.
Assignee: nhotta → putterman
Status: ASSIGNED → NEW
Component: Internationalization → Mail Back End
Keywords: intl
Summary: Can't delete/rename a folder with Japanese name containing a char with 2nd byte as 0x5b/0x5d/0x7c → Can't delete/rename a folder with name containing 0x5b/0x5d/0x7c
Please note that there are a lot of common used Japanese characters with 2nd byte as 0x5b/0x5d/0x7c.
reassigning to cavin.
Assignee: putterman → cavin
Status: NEW → ASSIGNED
Keywords: nsbeta1nsbeta1+
Priority: -- → P2
Target Milestone: --- → mozilla0.9.9
Target Milestone: mozilla0.9.9 → mozilla1.0
Per triage meeting, Shirley, can you try this bug on a build after 3-4, Scott thinks this could have been fixed by another bug fix. Please update the bug with your resulst.
Checked this with 03/06 build. It's fixed.
Status: ASSIGNED → RESOLVED
Closed: 23 years ago
Resolution: --- → FIXED
Marked it this as verified.
Status: RESOLVED → VERIFIED
This bug isn't completely fixed. �|ƒ|„|‰|Š|‹|Œ|�|Ž|�|�|‘|’|“|”|•|–|—|™|š|›|œ|�|ž|Ÿ|à|á|â|ã|ä|å|æ|ç|è|é|ê| �\ƒ\„\‰\Š\‹\Œ\�\Ž\�\�\‘\’\“\”\•\–\—\˜\™\š\›\œ\�\ž\Ÿ\à\á\â\ã\ä\å\æ\ç\è\é\ê\ Make a folder with name containing these characters. An invalid folder will be created.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
QA contact to kausmi. Thanks.
QA Contact: ji → kasumi
Son-san, Kasumi-san, could you change severity to critical, please. Although simple workaround is available, to avoid using error prone characters, mails loss occurs if such mail folder is moved or renamed. 2nd byte of "ƒ|„|‰|Š|‹|...etc." is 0x7C(|,ASCII pipe) in Shift_JIS. 2nd byte of "ƒ\„\‰\Š\‹\...etc." is 0x5C(\,ASCII backslash) in Shift_JIS. 0x7C case was mentioned by additional comment #10/#11/#13, but this case was not completely resolved in previous fix. 0x5C case is new in re-open by comment #18. Will this problem be resolved in 1.0.1 and 1.1?
tested on 2002-08-19-08-1.0 on Win XP Pro. JA Problem is occured but different than original one. Actualy, Folder stays in trash can but renamed and missing containing messages. This problem occures: 1. Folder name is having 0x5C, 0x7C, no problem 0x5D 2. Folder locates in Local folder, no problem in IMAP folder Message is missing is the big deal, therefore, change Severity to Critical
Severity: normal → critical
Kasumi-san, would you please change priority to P1? Could you change summary to "Mail loss on move/rename when mail folder name contains 0x5C, 0x7C as 2nd byte (MS Windows Environment, Local folder)" or more appropriate phrase? Since Japanese spell of Sony corporation is "ソニー", first 2 bytes are same as the first Japanese letter sample of 0x5C case in comment #20, we should not under estimate the impact of this problem. Netscape users do not care whether Mozilla is for development purpose or not. They belive it'a "Product". If Netscape 7.0 will be shipped without fix for this problem, many Japanese MS Windows users will not trust on Netscape/Mozilla any more. Furthermore, it may cause bad evaluation on Netscape 7 as Netscape 6.0 for any OS. I think this problem should be resolved before final release of Mozilla 1.0.1.
add it to 157673
Blocks: 157673
Wada-san: Sorry, I don't have athority to change priority.
> ------- Additional Comment #15 From esther@netscape.com 2002-03-06 18:52 ------- > > Per triage meeting, Shirley, can you try this bug on a build after 3-4, Scott > thinks this could have been fixed by another bug fix. Please update the bug > with your resulst. Anyone know which bug fixed part of the problem? We may take a look at the patch and find out what was missing. cc to putterman.
cc more people Some parts of this bug were once fixed by other bug that we cannot identify at this moment. But this bug is not completly fixed, there are remaining problems (comment #18 #21). Does anybody remember what is the other bug or which file we need to look at in order to fix the remaining cases?
Bug 41944 might be relevant to the issues being discussed here.
reassigning to naving
Assignee: cavin → naving
Status: REOPENED → NEW
I think delete/rename problem has already been fixed. There is still an issue remaining as described in comment #18. On start-up we show the disk name as prettyName. I have a fix for this problem.
Status: NEW → ASSIGNED
Attached patch proposed fix (obsolete) — Splinter Review
To fix invalid folder name when we restart after creating japenese folder I had to store the "folderName". Get/Set string property can be reused here. So when we start back up we look up for folderName and then set that as prettyName. Also I found out problems with CheckFolderIfExist where we were not passing proper disk name. So I fixed that. There was also some old unused code in nsMsgLocalMailFolder::Delete which I have removed. For moving local folders we use CopyFolderLocal now. Cavin, bienvenu, Can I get reviews ? thx.
Whiteboard: [ish1+]
After studying this patch for a long time, it looks like you're storing the actual name (as opposed to the mailbox name on disk) in the folder cache. If that's the case, it seems like this should really could be pushed all the way down to nsLocalMailFolder::SetName and and ::GetName, instead of making everyone who calls SetName and GetName also have to know to set/get the value from the folder cache. Is there some reason to keep two names around and have all the code maintain this? Or are there places where we really want to use the name on disk as the folder name and some places we don't? I looked through the code, and there are places where we call GetName (e.g., copying a local folder to another server) where I think we want to maintain the actual name instead, and I think making Set/GetName do the right thing would make this all transparent. To do this, I'd override ReadFromFolderCacheElem in nsLocalMailFolder.cpp, like nsImapMailFolder does, and make it get the folderName from the folderCache, convert it from utf8 to unicode, and put it in mName. And make WriteToFolderCacheElem write mName into the folderCache, after converting it to utf8. (there are advantages to using the folder cache as it was designed to be used - you let the folder cache code do its work, and you just use mName - no need to change any other code).
Overriding GetName/SetName for nsLocalMailFolder makes sense so that we can Set/GetStringProperty there. But name is something that doesn't change that frequently and we don't want to override the ReadFolderCacheElem/WriteToFolderCacheElem because they are called very often whenever you UpdateSummaryTotals and in other places. Storing/Retrieving unchanged information seems wasteful to me.
writing unchanged information is a noop in mork, so it's no big deal. It's one extra function call. Don't worry about it. That's not an issue.
forgive me if this is already taken care of, but this name has to survive folder compaction and summary file regeneration, so I would think the name has to be transferred using the dbtransfer info.
Instead of Get/SetProperty I have changed it to Read/Write from folder cache. The more part includes changing GetFolderCacheElemFromFileSpec to take a boolean createIfMissing. This is needed when we are setting a property when there is no cacheElement for a folder. CheckIfFolderExists checks for duplication based on folder names. There is an issue w/ prettyName being different from disk name and I have made it so that we check for both when we do CreateSubfolder. The other cases like rename and copy are also covered because disk copy/rename would fail. There is some clean-up coming from last patch - removing unused code. please review, thx.
Attachment #102983 - Attachment is obsolete: true
This looks better. However, I'm not sure that you want to only have the pretty name be the actual name (as opposed to the name on disk) - As I said before, I think you want to make Get and Set Name return the actual name, because there are places in the code where we call GetName where I think you really want the actual folder name, not the name on disk (for example, when you delete a folder, the name of the folder in the trash should be the japanese name). I know it's confusing, but there's no distinction between pretty name and actual name in the current code base - they used to be exactly the same, so the calling code could call GetName or GetPrettyName and it wouldn't matter. You've made them different, but it's a false distinction in the current code base, because the current code often calls GetName where I think you want it to call GetPrettyName. But I think we just want to change Get/Set Name, and leave the pretty name alone. Did you change PrettyName because it seemed like the right thing to do (which I agree, it does), or because it did the right thing in the code? As it is, I would just use Get/SetName because I don't think we need PrettyName, and avoiding introducing an extra distinction would be a good thing. Is there some reason we need PrettyName to be different from Name? Why do we ever need to create the folder cache element if it's missing? Now that we're storing the name in the dbfolderinfo, and we get it from there, and we always write out the info to the folder cache in nsMsgLocalMailFolder::WriteToFolderCacheElem, I don't think we ever need to force the folder cache elem to be created the way you're doing because nsMsgDBFolder::WriteToFolderCache will create the cache entry if it's missing, and we call WriteToFolderCache for every folder that exists.
PrettyName was there long before this bug. It is used to set localized name from messenger.properties if we have special name like Inbox and Inbox flag is set on the bug. So when we are setting name we should be using PrettyName. In certain cases we might directly want to SetName, but yes SetPrettyName functionality could be moved to SetName GetPrettyName and GetName do the same thing. Are you suggesting removing GetPrettyName, SetPrettyName, which I think is beyond the scope of the bug. Also when we are setting prettyName, we need to store in the cache and the db, so we need to create cacheElement in SetStringProperty. SetStringProperty is writing to the cache, so it should be creating an element if not there. It in itself is the right thing to to. I have seen how Writing to folder cache creates cacheElement but we cannot depend on that because SetStringProperty could be used separately before that happens and will fail causing problems. I made it completely generic by passing argument createIfMissing, still you are not agreeing!
For the third time, I'm saying use Get/SetName. To be as clear as I can possibly be, in your patch, where you're using Get/Set PrettyName, I'm saying use Get/SetName instead. I didn't say anything about removing it. I know PrettyName was there before! Did I say something that made you think I thought you invented it? It's been there since the 4.x days. My fear is that you don't really understand how the cache works, and how it should be used in conjunction with the folder object and the db. And that misunderstanding is what makes you want to add that createIfMissing. So I asked the question why you needed to do that. You then claimed that I was disagreeing with you still - I guess I can't ask questions, but since you didn't answer my question, that's no big loss, I guess. "SetStringProperty failing causes problems" does not really tell me very much. If you're forced to create the folder cache elem, that implies that we can't simply delete the folder cache and have everything still work. Does your patch work if you have a folder with a japanese name, shutdown, delete the folder cache, and start up again? If not, it's because you're not reading the actual name from the db. However, it looks to me like you are reading the folder name from the db in ReadDBFolderInfo, these lines: + nsXPIDLCString utf8Name; + folderInfo->GetFolderName(getter_Copies(utf8Name)); + if (!utf8Name.IsEmpty()) + mName = NS_ConvertUTF8toUCS2(utf8Name.get()); + so if the folder cache elem was missing, wouldn't we get the name from the db? If you can be a little bit more informative than SetStringProperty failing causing problems, that would be great. I spent some time looking through the code and your patch trying to guess what the problems might be, but instead of me trying to guess, perhaps you could just tell me the user scenario and what goes wrong in the code? Since at runtime, the folder object has the correct name, and when we shutdown, the db has the correct name, it's not clear to me why we need to force the creation of the folder cache elem. It's not like we should be reading the name from the folder cache elem after startup. Also, don't use an nsAutoString in an object like this. AutoStrings are stack based, so it doesn't make any sense to put them in objects that are allocated from the heap. + NS_IMETHOD GetFolderName(char **folderName); + NS_IMETHOD SetFolderName(const char *folderName); nsString m_boxName; + nsCAutoString m_folderName; Also, is RecursiveSetDeleteIsMoveToTrash used anymore, after you get rid of the two uses of it? If not, could you remove the method completely and finish the cleanup you've started? It would be great to actually get rid of some nsIMsgFolder methods. I could be wrong, but it looks to me like it's not used, because you're getting rid of two calls, and the remaining call is in RecursiveSetDeleteIsMoveToTrash itself. It also looks like if you get rid of that, you can also get rid of DeleteIsMoveToTrash itself (the methods and the member variable). Before doing any of this, of course, you should make sure that things like parent local folders moves them to the trash correctly, and that deleting them from the trash folder does the right thing as well.
I'm saying you *can't* use SetName. You have to use SetPrettyName because we want to set localized names for Inbox, trash etc. I know how the folderCache code works! But when we SetPrettyName we want to set the name in folder cache and in the db. So therefore I'm using SetStringProperty. At this point we haven't created cacheElement for this new folder. So were never setting this property "folderName" which was causing problems when you start-up. I agree WriteToFolderCache will create the element later, but I don't want to depend on that writing "folderName" in cache. We could crash before that happens. Does it make any sense ? Also I have taken care of the situation where folderCache doesn't exist, as you point out in those 4 lines. When you know what is going on, why ask un-necessary questions ? I can do more clean-ups for DeleteIsMoveToTrash and change folderName from nsCAutoString to nsString.
I have cleaned up deleteIsMoveToTrash and changed nsCAutoString to nsCString. I have tested these changes.
Attachment #103374 - Attachment is obsolete: true
I have cleaned up deleteIsMoveToTrash and changed nsCAutoString to nsCString. I have tested these changes.
Attachment #103644 - Attachment is obsolete: true
Sorry for the spam, attached wrong patch twice (have other changes in my tree). I have cleaned up deleteIsMoveToTrash and changed nsCAutoString to nsCString. I have tested these changes.
Attachment #103645 - Attachment is obsolete: true
I understand your point about SetPrettyName. There's code in SetPrettyName that does localizing of folder names, so now SetPrettyName actually does a translation and then sets the folder name to the translated name. And I see that you've admitted we don't need Get/SetPrettyName; you just don't want to fix it now. I guess it can get cleaned up later. I would have preferred not to have new calls to SetPrettyName get added to the code because that's just more crap to clean up when we get rid of Get/SetPrettyName but at this point, it's not worth the incredible hassle. I know these things don't bother you, but to me, having two attributes, name and prettyName, implies to me that there are two separate names, when, in fact, there aren't, and it leads to confusion. I still don't understand why you need to force the folder cache elem to get created because you're worried about crashing. If the code works fine when you start up without any folder cache at all (e.g., you delete it after shutting down mozilla), because the name gets read from the db, why does the same code not work when you start up with a single entry missing? Do you see my point? You write: "Also I have taken care of the situation where folderCache doesn't exist, as you point out in those 4 lines". Actually, that's my whole point. Those 4 lines handle the case where the folder cache entry doesn't exist, not that the whole folder cache doesn't exist. If the folder cache doesn't exist, you'll hit that code for every folder. If a single cache entry doesn't exist, you'll hit it for that folder. That's what I was trying to point out - that should handle the case of crashing before writing out the folder cache as well. If it doesn't, maybe there's something wrong with the folder cache in general that should be investigated. That's why I'd like to know why it doesn't work. With your approach, in your scenario, (correct me if I'm wrong), when you call SetStringProperty, you'll have a folder cache elem entry for the folder which *only* contains the name. All the other fields in the folder cache entry will be empty. If we commit the cache db (which we do quite frequently to make sure it doesn't get stale), and then crash before rewriting the folder cache completely, we'll have a folder cache entry that is incorrect, because it doesn't have all the other fields it should (those that get initialized through ReadFromFolderCacheElem, like the msg counts, etc). If we didn't have an incomplete folder cache entry, the next time we started up, we'd read all this info from the db. But since we would have an (incomplete) entry, we wouldn't try to read the info from the db, and we'll just display the default, empty, values for msg count, etc. The name will be correct, however.
the deleteIsMoveToTrash changes look fine, thx. Now folder::Delete really does a delete for each kind of folder, which is great.
Attached patch patchSplinter Review
point taken about folder cache, leaving entry creation to WriteFolderCache...
Attachment #103647 - Attachment is obsolete: true
Comment on attachment 103661 [details] [diff] [review] patch sr=bienvenu,thx.
Attachment #103661 - Flags: superreview+
Comment on attachment 103661 [details] [diff] [review] patch r=cavin.
Attachment #103661 - Flags: review+
fix checked on ishmail branch.
Whiteboard: [ish1+] → [ish1+][fixedish1]
I've tested with trunc build id: 2002102704 (win2-talkback.zip/Win-ME). [Test Cases] folder name = 0x835C in Shift_JIS ( katakana "so" in Japanese, 2nd byte is \ ) folder name = 0x837C in Shift_JIS ( katakana "po" in Japanese, 2nd byte is | ) (1) Creation of mail folder -> OK ( No file whose file name is "po" or "so" of Katakana was created ) (2) Copy/Move/Delete mails -> OK (mail count is correct) (3) Creation of sub-folder -> OK (4) Move sub-folder -> OK (5) Delete sub-folder -> OK Patch worked very very well in Windows ME environment. Mail data integurity exposure problem has been resolved completely. Thanks a lot. (6) Move a sub-folder to another folder where same sub-folder already exist -> OK ( No action was taken ) -> (Problem-1) No warning message was issued (7) Mail move/copy to a sub-folder of a folder(named "so" or "po" of Katakana) -> (Problem-2) sub-folders are not expanded in context menu Problem-2 occurs even when sub-folder's name consists of alphabets only. Problem-1 is a minor problem. Problem-2 seems to be a major problem. Should I report above 2 problems as new bug?
Test report(Continued) When Mozilla was restarted, mail folder name becomes file name instead of original folder name (trunc-2002102704/Win-Me). For folder name of "so" or "po" of Katakana, 3db4214a or 3db4216a was the real file name. By the way, how can I know original folder name from mail folder files? Can we import mail folders on other mailer with original folder name?
The fix has not been checked on trunk yet.
I have tested same test cases in comment 49 with trunc build id: 20021028, win32-talkback, Windows XP with NTFS folder name = ( katakana "so" in Japanese ) folder name = ( katakana "po" in Japanese ) (I don't know character code of Japanese in Win/XP NTFS) > (1) Creation of mail folder > -> OK ( No file whose file name is "po" or "so" of Katakana was created ) > (2) Copy/Move/Delete mails -> OK (mail count is correct) > (3) Creation of sub-folder -> OK > (4) Move sub-folder -> OK > (5) Delete sub-folder -> OK (1)-(5): I get same results. But I found problems. 1. Create following folders. Inbox +"so" --- katakana "so" |+111 --- subfolder 2. After restarting, I have following folders: Inbox +3db4214a --- renamed from katakana "so" |+111 3. Creation of mail folder named katakana "so" in Inbox again. Then I get ... Inbox +3db4214a --- (a) renamed from katakana "so" |+111 --- (b) subfolder +"so" --- (c) katakana "so" |+111 --- (d) subfolder Duplicated mail folders katakana "so" and 111 appear, and those folders have same mails with original folders, i.e.. (a) and (c) have same mails, (b) and (d) have same mails. 4. After restarting, duplicated mail folders disappear. Inbox +3db4214a --- (a) renamed from katakana "so" |+111 --- (b) subfolder - Renamed mail folders are not acceptable for me. - Duplicated mail folders is a minor bug, but it will make confusion to users.
> The fix has not been checked on trunk yet. Oops! sorry for my spam :(
Is there any build where this bug is fixed?
verified on 11-01 build and changed Status Whiteboard
Whiteboard: [ish1+][fixedish1] → [ish1+][verifiedish1]
fixed
Status: ASSIGNED → RESOLVED
Closed: 23 years ago22 years ago
Resolution: --- → FIXED
I want to test this bug. Which build should be used?
Blocks: 178570
The check in was made at 5:30pm on Nnov 5 US time. So you download a trunk build for 2002-11-06 or later, it should contain the fix for this bug.
2002110610 on WinXP Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2b) Gecko/20021106 It has still problems. problem 1: (1) Create a folder and copy some messages to it. (2) Remove the folder. (3) Create a folder with same name. Then, at first, wrong message counter is displayed. problem 2: (1) Create a folder named KATAKANA "so". The file name becomes "3db4214a". Copy some messages to it. (2) Rename the folder name to KATAKANA "po". The file name becomes "3db4216a". (3) Create a folder named KATAKANA "so" again. Then folder "so" and "po" share same messages. New file "3db4214a" is created, but no .msf file. (4) After restart, folder "po" is renamed to "so". It means that two folders, named "so", are appeared.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
problem 3: (1) Create a folder named "/". Then, mozilla is stalled.
All the 3 problems that you listed are known problems. I just attempted to fix the problem where you will see disk name for certain japanese character folder names when you restart. That problem is fixed. Problem 1 is because I think it gets count from the cache. we should remove cache entry when deleting a folder. Relevant bugs for problem 2 are bug 65303 bug 64463 and I think there is a bug for problem 3 as well, can't find it. Problem 2 is happening because we leak rdfResource. closing this bug. please file separate bug for each problem.
Status: REOPENED → RESOLVED
Closed: 22 years ago22 years ago
Resolution: --- → FIXED
verified on 2002-11-07-08-1.0 Local and IMAP folders butthis fix causes another problem which is that folder name turn to be garbage in local folder. I will report sepalately. Koike-san: I verified. If you will find any problem, please reopen.
Status: RESOLVED → VERIFIED
(Q1) Cause of problem 1 seems to be same as bug 65303 bug 64463, problem 2. Is adding comment to bug 65303 sufficient? Or should we open a new bug for problem 1? (Q2) I found bug 115091 which is freeze problem when mail folder contains "#". Is this a relevant bug to problem 3 ? Anyway, now we Japanese can use Japanese characters in folder name without a fear of mail loss. Navin san, thanks a lot for your fix.
Depends on: 180372
No longer blocks: 157673
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: