Closed Bug 154000 Opened 22 years ago Closed 22 years ago

Can't move non-ascii subfolder on IMAP

Categories

(MailNews Core :: Internationalization, defect)

Product:
MailNews Core
Component:
Internationalization
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED
Milestone:
mozilla1.0.1

People

(Reporter: jeesun, Assigned: naving)

References

Details

(Keywords: dataloss, intl, Whiteboard: [adt1 rtm] [ETA 06/28])

Attachments

(1 file, 5 obsolete files)

proposed fix
2.49 KB, patch
cavin
: review+
Bienvenu
: superreview+
chofmann
: approval+
Details | Diff | Splinter Review 
Can't move non-ascii subfolder under special folders on IMAP
Platform: JA windows XP
Input: JA chars

Steps:
1. On IMAP account, create a non-ascii folder and subfolder under it.
2. Move them (both at the same time by highlighing the 1st level folder) by
dragging and dropping them to Sent or Drafts folder.
3. Both will appear under Sent folder for example, but when you click on the 2nd
level folder, you'll get an error: The mail server responded: Mailbox does not exist

English folders work fine.
This seems related to bug 153982
Keywords: intl
btw: this error can be reproduced with any folder on the IMAP tree (not only
with special folders). Try to drag the newly created non-ascii subfolder to the
top of the tree (jeesun@netscape.com) or to any ascii/non-ascii folder of the
first level: you'll get the same error- seems that the copy/move of the
non-ascii subfolder is not allowed.
You're right, Marina. Change the summury accordingly.
Summary: Can't move non-ascii subfolder under special folders on IMAP → Can't move non-ascii subfolder on IMAP
changing qa contact.Thanks!
QA Contact: marina → jeesun
This is dataloss. Is there anyway to recover the data?
Nominating for nsbeta1. Reassign to naving, cc to putterman
Assignee: nhotta → naving
Keywords: dataloss, nsbeta1
I can reproduce this with Latin1 characters.

parent folder "ссс", subfolder "ррр".

1) In nsImapMailFolder::RenameSubFolders, 
http://lxr.mozilla.org/seamonkey/source/mailnews/imap/src/nsImapMailFolder.cpp#6905

6905      rv = AddSubfolderWithPath(&utf7LeafName, dbFileSpec,
getter_AddRefs(child));
The argument utf7LeafName contains non UTF-7 after the call. Then the following
call to append to CString fails because utf7LeafName contains non ASCII.

2) In nsImapMailFolder::AddSubfolderWithPath,
http://lxr.mozilla.org/seamonkey/source/mailnews/imap/src/nsImapMailFolder.cpp#324
323   uri.Append(*name);
324   char* uriStr = ToNewCString(uri);

this doesn't seem to be right since 'name' may be non ASCII

I got assersions for both places with non ASCII subfolder name.

Naving, could you take a look at this?


>I can reproduce this with Latin1 characters.
>parent folder "ссс", subfolder "ррр".
Parent folder does not have to be non ASCII, subfolder has to be non ASCII.

this is an important intl functionality, it is not a regression from 6.2 (is
reproducable in the shipped versoins)
Attached patch Convert subfolder to UTF-7. (obsolete) — — Splinter Review 
The dataloss problem is fixed with this patch. But I still see the assersion
for AddSubfolderWithPath using ToNewCstring.
I suggest we take this as nsbeta1+ and mark it as [adt1] since this could be a
huge data loss (the loss of a whole folder of messages) . Since I am not the
engineer manager of naving, I will leave that to his manager to mark that.
Blocks: 141008
N6.2 have this problem too. So this is not a regression frrom 6.2 to 7.0
accepting. working on a fix.
Status: NEW → ASSIGNED
If they are really losing their folder then this is a pretty critical bug.
Keywords: nsbeta1 → nsbeta1+
Whiteboard: [adt1 rtm]
Scott, please also look at bug 153982, similar problem also happens for POP.
Attached patch proposed fix (obsolete) — — Splinter Review 
I made some changes to nhotta's last patch. We need to use modutf7 for online
name.
Regarding assertion the problem is in AddSubfolderWithPath where it does
ToNewCString. It should actually convert name to modutf7 when constructing
uriStr. I think this patch is ok to fix this bug.
Cavin, David, Can I get reviews ? thx.
I think rdf uri's have to be in utf8 format so we cannot do what I suggested to
fix the assertion. 
Comment on attachment 89115 [details] [diff] [review]
proposed fix

r=cavin.
This fixes the assertion for me in AddSubfolderWithPath. still investigating if
this is correct

@@ -321,7 +321,7 @@
   uri.Append(PRUnichar('/'));
 
   uri.Append(*name);
-  char* uriStr = ToNewCString(uri);
+  char* uriStr = nsCRT::strdup(NS_ConvertUCS2toUTF8(uri.get()).get());
   if (uriStr == nsnull) 
     return NS_ERROR_OUT_OF_MEMORY;
Attached patch complete patch fixing assertion (obsolete) — — Splinter Review 

        Attachment #89115 -
        Attachment is obsolete: true

    
  
Blocks: 143047
Whiteboard: [adt1 rtm] → [adt1 rtm] [ETA 06/27]
Target Milestone: --- → mozilla1.0.1

    
    

    
  
Comment on attachment 89115 [details] [diff] [review]
proposed fix

sr=bienvenu

        Attachment #89115 -
        Flags: superreview+

        Attachment #89115 -
        Flags: review+

    
    

    
  
fixed on trunk
Status: ASSIGNED → RESOLVED
Closed: 22 years ago
Keywords: adt1.0.1
Resolution: --- → FIXED

    
    

    
  
jeesun - cna you pls verify this fix on the trunk, then  mark it as resolved?
thanks!

    
    

    
  
I'll pick up tomorrow's trunk build and verify this.

    
    

    
  
Reopen this. Try this slightly different scenario. I'm still experiencing data
loss. See the attached screenshot.
1. Create a non-ascii folder and a subfolder under it (ゆき、　ゆき２）
2. Move both to a 1st level folder (ifolder)
3. Then create ゆき３ under ゆき２
4. Notice the tree structure looks weird (The original ゆき２ is grayed-out and
the same folder was created again. See the screenshot.)
5. Move the whole ゆき１，２，３　folders back to the top of the tree, out of
ifolder
6. ゆき３ disappeared
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  

      
      
      
      

        Attached image
          
        non-ascii folder screenshot (obsolete)
        — 
      

    


  
Please see this page in Shift_JIS

    
    

    
  
Please create a reproducible test case for your comment #26, so everyone can
test/debug.

    
    

    
  
Jeesun, plesae use Latin1 characters (e.g. á) for your test case (when the
problem is generic for non ASCII characters), so pepole who do not have Japanese
system support can reproduce and debug, thanks.


    
    

    
  
Sure! This is reproducible using latin-1 chars too. Just use latin-1 chars where
I mention JA chars.
My intention was to give you an example to describe the problem and reproducible
steps in more detail. 

    
    

    
  
I don't know what is latin-1 and what is japanese. Can you give a list of
reproducible steps with folder names in latin-1.


    
    

    
  
1. Create a latin-1 folder, for example, tàmü and a subfolder, tàmü2 under it 
2. Move both to a 1st level folder, say its name is ifolder.
3. Then create yet another subfolder, tàmü3 under tàmü2
4. Notice the tree structure looks weird (The original tàmü2 is grayed-out and
the same folder was created again.)
5. Move the whole tàmü, tàmü2, tàmü3 folders back to the top of the tree, out of
ifolder
6. tàmü3 disappeared

    
    

    
  
removing the nomination because this bug has been reopened.
Keywords: adt1.0.1

    
    

    
  
working on a fix. 
Status: REOPENED → ASSIGNED

    
    

    
  
I have a fix for problem described in comment #32. But I am seeing other
problems where LSUB command doesn't tell us all the children so when we
*restart* tàmü3 disappears. This is relevant portion of the imap log for the
last rename command
when doing comment #32. Remember we do lsub command before we do rename. Relevant
code to look at is RenameHierarchyByHand in nsImapProtocol. David, what do you 
think ?

2512[4ffa200]: nsmail-2:S-ifolder/t&AOA-m&APw-:SendData: 6 lsub ""
"ifolder/t&AOA-m&APw-/*"

2512[4ffa200]: nsmail-2:S-ifolder/t&AOA-m&APw-:CreateNewLineFromSocket: * LSUB
(\HasNoChildren) "/" ifolder/t&AOA-m&APw-/t&AOA-m&APw-1/t&AOA-m&APw-2

2512[4ffa200]: nsmail-2:S-ifolder/t&AOA-m&APw-:CreateNewLineFromSocket: 6 OK
Completed

2512[4ffa200]: nsmail-2:S-ifolder/t&AOA-m&APw-:SendData: 7 close

2512[4ffa200]: nsmail-2:S-ifolder/t&AOA-m&APw-:CreateNewLineFromSocket: 7 OK
Completed

2512[4ffa200]: nsmail-2:A:SendData: 8 rename "ifolder/t&AOA-m&APw-" "t&AOA-m&APw-"

2512[4ffa200]: nsmail-2:A:CreateNewLineFromSocket: 8 OK Completed

2512[4ffa200]: nsmail-2:A:SendData: 9 subscribe "t&AOA-m&APw-"

2512[4ffa200]: nsmail-2:A:CreateNewLineFromSocket: 9 OK Completed

2512[4ffa200]: nsmail-2:A:SendData: 10 unsubscribe "ifolder/t&AOA-m&APw-"

2512[4ffa200]: nsmail-2:A:CreateNewLineFromSocket: 10 OK Completed

2512[4ffa200]: nsmail-2:A:SendData: 11 subscribe
"t&AOA-m&APw-w-/t&-AOA-m&-APw-1/t&-AOA-m&-APw-2"

2512[4ffa200]: nsmail-2:A:CreateNewLineFromSocket: 11 NO Mailbox does not exist

2512[4ffa200]: nsmail-2:A:SendData: 12 unsubscribe
"ifolder/t&-AOA-m&-APw-/t&-AOA-m&-APw-1/t&-AOA-m&-APw-2"

2512[4ffa200]: nsmail-2:A:CreateNewLineFromSocket: 12 OK Completed

2716[50e0c50]: nsmail-2:NA:CreateNewLineFromSocket: * OK dredd.mcom.com IMAP4
service (Netscape Messaging Server 4.15 Patch 4 (built Dec  7 2000))

2716[50e0c50]: nsmail-2:NA:SendData: 1 authenticate plain

2716[50e0c50]: nsmail-2:NA:CreateNewLineFromSocket: + 

2716[50e0c50]: nsmail-2:NA:SendData: Logging suppressed for this command (it
probably contained authentication information)
2716[50e0c50]: nsmail-2:NA:CreateNewLineFromSocket: 1 OK User logged in (no
protection)

2716[50e0c50]: nsmail-2:A:SendData: 2 XSERVERINFO MANAGEACCOUNTURL
MANAGELISTSURL MANAGEFILTERSURL

2716[50e0c50]: nsmail-2:A:CreateNewLineFromSocket: * XSERVERINFO
MANAGEACCOUNTURL {60}

2716[50e0c50]: nsmail-2:A:CreateNewLineFromSocket:
http://naving@dredd.mcom.com:4444/bin/user/admin/bin/enduser MANAGELISTSURL NIL
MANAGEFILTERSURL NIL

2716[50e0c50]: nsmail-2:A:CreateNewLineFromSocket: 2 OK Completed



    
    

    
  
If I try to break up the hierarchy and subscribe for each children folder then 
also it doesn't fix it. cc jgmyers for input as well. 

3040[51627b0]: nsmail-2:S-ifolder/t&AOA-m&APw-:SendData: 6 
lsub "" "ifolder/t&AOA-m&APw-/*"

3040[51627b0]: nsmail-2:S-ifolder/t&AOA-m&APw-:CreateNewLineFromSocket: * LSUB 
(\HasNoChildren) "/" ifolder/t&AOA-m&APw-/t&AOA-m&APw-2/t&AOA-m&APw-3

3040[51627b0]: nsmail-2:S-ifolder/t&AOA-m&APw-:CreateNewLineFromSocket: 6 OK 
Completed

3040[51627b0]: nsmail-2:S-ifolder/t&AOA-m&APw-:SendData: 7 close

3040[51627b0]: nsmail-2:S-ifolder/t&AOA-m&APw-:CreateNewLineFromSocket: 7 OK 
Completed

3040[51627b0]: nsmail-2:A:SendData: 8 rename "ifolder/t&AOA-m&APw-" "t&AOA-
m&APw-"

3040[51627b0]: nsmail-2:A:CreateNewLineFromSocket: 8 OK Completed

3040[51627b0]: nsmail-2:A:SendData: 9 subscribe "t&AOA-m&APw-"

3040[51627b0]: nsmail-2:A:CreateNewLineFromSocket: 9 OK Completed

3040[51627b0]: nsmail-2:A:SendData: 10 unsubscribe "ifolder/t&AOA-m&APw-"

3040[51627b0]: nsmail-2:A:CreateNewLineFromSocket: 10 OK Completed

3040[51627b0]: nsmail-2:A:SendData: 11 subscribe "t&AOA-m&APw-w-/t&-AOA-m&-APw-
2"

3040[51627b0]: nsmail-2:A:CreateNewLineFromSocket: 11 NO Mailbox does not exist

3040[51627b0]: nsmail-2:A:SendData: 12 unsubscribe "ifolder/t&-AOA-m&-APw-/t&-
AOA-m&-APw-2"

3040[51627b0]: nsmail-2:A:CreateNewLineFromSocket: 12 OK Completed

3040[51627b0]: nsmail-2:A:SendData: 13 subscribe "t&AOA-m&APw-w-/t&-AOA-m&-APw-
2/t&-AOA-m&-APw-3"

3040[51627b0]: nsmail-2:A:CreateNewLineFromSocket: 13 NO Mailbox does not exist

3040[51627b0]: nsmail-2:A:SendData: 14 unsubscribe "ifolder/t&-AOA-m&-APw-/t&-
AOA-m&-APw-2/t&-AOA-m&-APw-3"

3040[51627b0]: nsmail-2:A:CreateNewLineFromSocket: 14 OK Completed

2624[518f7b0]: nsmail-2:A:SendData: 12 select "t&AOA-m&APw-"

2624[518f7b0]: nsmail-2:A:CreateNewLineFromSocket: * FLAGS (\Answered \Flagged 
\Draft \Deleted \Seen)

2624[518f7b0]: nsmail-2:A:CreateNewLineFromSocket: * OK [PERMANENTFLAGS 
(\Answered \Flagged \Draft \Deleted \Seen \*)]  

2624[518f7b0]: nsmail-2:A:CreateNewLineFromSocket: * 0 EXISTS

2624[518f7b0]: nsmail-2:A:CreateNewLineFromSocket: * 0 RECENT

2624[518f7b0]: nsmail-2:A:CreateNewLineFromSocket: * OK [UIDVALIDITY 
1025145230]  

2624[518f7b0]: nsmail-2:A:CreateNewLineFromSocket: 12 OK [READ-WRITE] Completed


    
    

    
  


  
This patch fixes the bug but there are some problems as described in my earlier

comments. AddSubfolderWithPath should actually take char* instead of
nsAutoString if we want to avoid converting back and forth from unicode to utf7

and vice-versa. But we can do that in a separate bug.

    
    

    
  
this subscribe failed:
2512[4ffa200]: nsmail-2:A:SendData: 11 subscribe
"t&AOA-m&APw-w-/t&-AOA-m&-APw-1/t&-AOA-m&-APw-2"

2512[4ffa200]: nsmail-2:A:CreateNewLineFromSocket: 11 NO Mailbox does not exist

and here:
3040[51627b0]: nsmail-2:A:SendData: 11 subscribe "t&AOA-m&APw-w-/t&-AOA-m&-APw-
2"

3040[51627b0]: nsmail-2:A:CreateNewLineFromSocket: 11 NO Mailbox does not exist

presumably, that's the cause of the problem. I can't tell if your patch fixes
that problem or not.

    
    

    
  
The last patch doesn't fix the problem of tàmü3 disappearing on restart. 

>This patch fixes the bug but there are some problems as described in my earlier
>comments.

Actually if I try the same thing with ascii folder names it works fine. So this 
problem (tàmü3 disappearing on restart) appears to be a *server issue* because
the server doesn't tell us the children correctly on doing lsub command. 
For a comparison please look at comment #35 and this comment, thx. 


2624[4c02a10]: nsmail-2:S-ifolder/A:SendData: 7 lsub "" "ifolder/A/*"

2624[4c02a10]: nsmail-2:S-ifolder/A:CreateNewLineFromSocket: * LSUB 
(\HasChildren) "/" ifolder/A/B

2624[4c02a10]: nsmail-2:S-ifolder/A:CreateNewLineFromSocket: * LSUB 
(\HasNoChildren) "/" ifolder/A/B/C

2624[4c02a10]: nsmail-2:S-ifolder/A:CreateNewLineFromSocket: 7 OK Completed

2624[4c02a10]: nsmail-2:S-ifolder/A:SendData: 8 close

2624[4c02a10]: nsmail-2:S-ifolder/A:CreateNewLineFromSocket: 8 OK Completed

2624[4c02a10]: nsmail-2:A:SendData: 9 rename "ifolder/A" "A"

2624[4c02a10]: nsmail-2:A:CreateNewLineFromSocket: 9 OK Completed

2624[4c02a10]: nsmail-2:A:SendData: 10 subscribe "A"

2624[4c02a10]: nsmail-2:A:CreateNewLineFromSocket: 10 OK Completed

2624[4c02a10]: nsmail-2:A:SendData: 11 unsubscribe "ifolder/A"

2624[4c02a10]: nsmail-2:A:CreateNewLineFromSocket: 11 OK Completed

2624[4c02a10]: nsmail-2:A:SendData: 12 subscribe "A/B"

2624[4c02a10]: nsmail-2:A:CreateNewLineFromSocket: 12 OK Completed

2624[4c02a10]: nsmail-2:A:SendData: 13 unsubscribe "ifolder/A/B"

2624[4c02a10]: nsmail-2:A:CreateNewLineFromSocket: 13 OK Completed

2624[4c02a10]: nsmail-2:A:SendData: 14 subscribe "A/B/C"

2624[4c02a10]: nsmail-2:A:CreateNewLineFromSocket: 14 OK Completed

2624[4c02a10]: nsmail-2:A:SendData: 15 unsubscribe "ifolder/A/B/C"

2624[4c02a10]: nsmail-2:A:CreateNewLineFromSocket: 15 OK Completed

2980[4d68730]: nsmail-2:NA:CreateNewLineFromSocket: * OK dredd.mcom.com IMAP4 
service (Netscape Messaging Server 4.15 Patch 4 (built Dec  7 2000))

2980[4d68730]: nsmail-2:NA:SendData: 1 authenticate plain

    
    

    
  
Cc to taka for the server problem.

    
    

    
  
Navin, did you look in the subscribe UI and see if your subscribe was
successful? It looks to me from the log that it failed.

    
    

    
  
From comment 35:

2512[4ffa200]: nsmail-2:A:SendData: 11 subscribe
"t&AOA-m&APw-w-/t&-AOA-m&-APw-1/t&-AOA-m&-APw-2"

This appears to have been garbled by the client.  Shouldn't it instead be:

11 subscribe "t&AOA-m&APw-w-/t&-AOA-m&-APw-1/t&-AOA-m&APw-2"

to match what the server reported in the previous LSUB response?


    
    

    
  
yes, subscribe UI failed. Good point, jgmyers. 

    
    

    
  
a point that I tried to make twice :-(

    
    

    
  
The difference was so subtle I couldn't make out (until it was explicitly 
pointed out). Sorry about that!

patch upcoming.

    
    

    
  
sorry, I should have been more clear - the subscribe should not fail, and the
reason it failed was that we were subscribing to a folder that didn't exist
because the name was wrong - but that imap mod-utf7 is a bear to read.

    
    

    
  
The cause of garbled folder name could be in charset convertor.
Just my guess.

    
    

    
  

      
      
      
      

        Attached patch
          
          
        proposed fix
        — 
          — Splinter Review
      

    


  
There are two parts to the fix. One is to pass unicode string converted from
mod-utf7 to AddSubfolderWithPath. The next part is in imapProtocol code where
the server is giving us names in modutf7 so we don't need to convert the server
name again to mod-utf7. That was causing subscribe to fail.

Cavin, David, Can I get reviews ? thx.

        Attachment #89090 -
        Attachment is obsolete: true

        Attachment #89136 -
        Attachment is obsolete: true

        Attachment #89313 -
        Attachment is obsolete: true

        Attachment #89354 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 89444 [details] [diff] [review]
proposed fix

sr=bienvenu - however, this does require a lot of testing of mod-utf7 folder
names (e.g., A&B), creating, renaming, deleting, subscribe/unsubscribing, etc.

        Attachment #89444 -
        Flags: superreview+

    
    

    
  
This code change will just affect renaming/deleting/moving imap folders and 
that is all it would affect. I have tested these three, looks ok. 

    
    

    
  
*** Bug 154446 has been marked as a duplicate of this bug. ***

    
    

    
  
Comment on attachment 89444 [details] [diff] [review]
proposed fix

r=cavin.

        Attachment #89444 -
        Flags: review+

    
    

    
  
If I am reading comments correctly, there is a bug in charset
convertor from Modified UTF-7 to UCS-2.  We should double check
the convertor works okay and open a new bug against it if it doesn't.



    
    

    
  
fixed on trunk. 
Status: ASSIGNED → RESOLVED
Closed: 22 years ago → 22 years ago
Resolution: --- → FIXED

    
    

    
  
yes, it does seem like the round trip might not quite be working from imap
mod-utf7 -> ucs2 -> imap mod-utf7, unless somewhere we're just converting to
utf7 and not imap mod-utf7

    
    

    
  
we were passing modutf7 to CreateUtf7ConvertedString when it was expecting 8 
bit ascii, so I think it is ok now unless we have some other places doing the 
same thing. 

    
    

    
  
Verified on 0628 trunk build.
Status: RESOLVED → VERIFIED

    
  
Keywords: adt1.0.1, 
          
            mozilla1.0.1

    
    

    
  
adt1.0.1+ (on ADT's behalf) approval for checkin to the 1.0 branch. pls check
this in asap, then add the "fixed1.0.1" keyword.
Keywords: adt1.0.1 → adt1.0.1+
Whiteboard: [adt1 rtm] [ETA 06/27] → [adt1 rtm] [ETA 06/28]

    
    

    
  
Comment on attachment 89444 [details] [diff] [review]
proposed fix

a=chofmann for 1.0.1 add the fixed1.0.1 keyword after checking in to the
branch.

        Attachment #89444 -
        Flags: approval+

    
  
Keywords: mozilla1.0.1 → mozilla1.0.1+

    
    

    
  
fixed on branch.
Keywords: mozilla1.0.1+ → fixed1.0.1

    
    

    
  
Verified on branch build.
Keywords: verified1.0.1
Product: MailNews → Core
Product: Core → MailNews Core

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: