474468 - Drag&drop copy of e-mail between POP3 account to Local Folders aborts silently

Status: RESOLVED INCOMPLETE

(Thunderbird :: Folder and Message Lists, defect)

Description

[paladin]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008122010 Iceweasel/3.0.5 (Debian-3.0.5-1)
Build Identifier: version 2.0.0.19 (20081209)

Test 1:

Copy COMPLEX folder structure from a POP3 account to IMAP server. The total
size of that complex folder structure is 2.4GB and each sub-folder can have
from 1kB to 300MB of e-mail. The complex directory structure is created in the
IMAP server and some of the e-mails are copied. Nevertheless, the copying
process aborts silently. Many e-mails are missing in the destination folders
and most folders are empty. I'm not able to understand what is the pattern of
which e-mails are copied and which are missing.

Test 2:

Copy the same COMPLEX folder structure from a POP3 account to Local Folders. I
obtain the same result as before, confirming that the problem is not in the
copy to IMAP, but on the copy itself.


Reproducible: Always

Steps to Reproduce:
1. Click top level folder and drag it to destination folder while pressing Ctrl

Actual Results:  
Not all mails were copied, some of the destination folders were even empty.

Expected Results:  
Exact copy of the original folder tree.

Original test was done on a Linux x86_64 machine, but I've also tested under a 32 bit Windows XP virtual machine with the same results.

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Reproducible: Always
> Steps to Reproduce:
> 1. Click top level folder and drag it to destination folder while pressing Ctrl

Following is copy of Bug 471162 Comment #5.
> Following is Seamonkey's help inherited from Mozilla/Mozilla App Suite).
> AFAIK, developers of Tb didn't change this basic design/implementation.
> Where can I see document which describes that "Ctrl+Drag&Drop of mail folder"
is "folder copy on Tb"?
>
> > Moving or Copying a Folder
> >
> > You can copy a folder and its contents to another mail account,
> > or move a folder within the same mail account.
> > To move or copy a folder, begin from the Mail window:
> >    1. Select the folder you want to move or copy.
> >    2. Do one of the following:
> >       *  To move the folder under another folder within the same account, drag
> >            the folder over the name of the other folder. The folder you moved
> >            becomes a subfolder of the other folder.
> >       *  To copy the folder to another account, drag the folder over the name
> >            of another account.
> >       *  To copy the folder under another folder in another account, drag the
> >            folder over the name of another folder in another account.
> >            The folder you copied becomes a subfolder of the other folder.

See Bug 471336 for incorrect behaviour when "Ctrl+Drag&Drop of mail folder".
> Bug 471336 ctrl key activated drag and drop folders within folder pane has wrong icon

When did you release "Ctrl"?
1. Press Ctrl, 2. Drag with pressing Ctrl, 3. Drop(mouse up), 4. Release Ctrl ?

Comment 2

[paladin]

Yeap! In fact, I did both with the same result!

Comment 3

[WADA:World Anti-bad-Duping Agency]

Can you reproduce problem with "Drag&Drop"(not Ctrl+Drag&Drop), with copy from Local to Local?

Comment 4

[paladin]

It seems I only did the "Drag&Drop"(not Ctrl+Drag&Drop) when copying to IMAP. It copied almost everything, only 600MB are missing. This is what failed to be copied:

1. Contents of folders which have a "/" in the name.
2. A third level folder named "#".
3. A fourth level folder named "B".
4. There is a third level folder, which contains 52 sub-folders. From the 52 sub-folders something like 1/3 is empty in the target.

Now the strange thing. I now tried to erase the target folder tree by selecting "delete" when right-clicking and it didn't work. I then used Drag&Drop it to Trash and it still didn't remove the folder tree (even though there was I/O). It was necessary to remove everything manually.

Tried the Drag&Drop again, with similar results: the same folders are missing in target (there is another missing which I didn't notice before, but I'm not 100% sure).

Is there any way to increase verbosity of this copy process?

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> It seems I only did the "Drag&Drop"(not Ctrl+Drag&Drop) when copying to IMAP.

Does it mean that you DID do "Ctrl+Drag&Drop" always in test of copy from local(POP3) to local(Local Folders) case? 

> It copied almost everything, only 600MB are missing. This is what failed to be copied:

600MB = (What size of which) - (What size of which)?
Please note that "same file size" has meaning only when;
  - Copy of local mail folder to local mail folder,
    which is already compacted(or compacted after copy).
  - File size(data size) of file for a mail folder(xxx, not xxx.msf)   
Even when local to local & compacted, "disk space for file" can be different unless both files are placed on same physical drive with same file system.
Do you compare (What size of which of IMAP) with (What size of which of Local)?

> 1. Contents of folders which have a "/" in the name.
> 2. A third level folder named "#".

(1) "/" is special. 
(1a) "/" is illegal local file name character, so escaped(hash value is used)
     when local mail older file is created.
     xxx/yyy => xxx<hash_value> is used as local file name
(1b) "/" is folder hierarchy delimitor of many IMAP server. In this case,
     create request of xxx/yyy via IMAP means "xxx" and "yyy under xxx".
(2) "#" is special. (indicator of "hash" in URL)
    Tb still has "#" related issue especially on IMAP folder.
See Dependency tree for meta Bug 124287.
	
It looks that your problem in "Local to IMAP" case is special character related problem. Test with minimum test case, getting IMAP log, and open separate bug after check of real IMAP level flow, please.

Comment 6

[paladin]

I can only check this next week.
Sorry.

Comment 7

[paladin]

Sorry for misleading you again by talking about IMAP. The tests I reported for _this_ bug were only done from POP3 to Local Folders.

Yes, I always did "Ctrl+Drag&Drop".

When I say "600MB missing" I mean that there are 600MB missing in the destination folder. That is, 600MB = POP3 Account/.../Inbox.sbd - Local Folders/.../Inbox.sbd. Yes, they have been compacted.

I've tried again (now using "Drag&Drop") and renamed all "strange" folder names and obtained the same result. I can see empty and missing folders on Local Folders after copy.

I'm attaching two files, with the folder structure and respective sizes on the POP3 account and local folders. You can diff them to see what I mean.

Comment 8

[paladin]

Attachment: Folder structure in POP3 Account

Comment 9

[paladin]

Attachment: Local Folders structure

Comment 10

[paladin]

Funny thing: after doing this copy I'm unable to delete the destination folder. I had to do it through the OS.

Comment 11

[WADA:World Anti-bad-Duping Agency]

> 4,0K /Inbox.sbd/Level 1/Level 2/Level 3/Level 4

You defined the account as movemail account?
> http://www.mozilla.org/mailnews/movemail/index.html
Or you did define Tb's mail folder named "/Level 1/Level 2/Level 3/Level 4"?
Is was "Saved Search Folder(Virtaul Folder)"?

> Build Identifier: version 2.0.0.19 (20081209)

What is obtained by "copy folder location" of folder corresponds to "/Level 4"?
  - Context menu of mail folder, "Copy Folder Location", Paste at text editor.
Check "/Level 4" of both Drag&Drop source account and Drag&Drop target account.

Comment 12

[WADA:World Anti-bad-Duping Agency]

By the way, following problem has been ruled out, because "Drag&Drop" and "Ctrl+Drag&Drop" produce same result.
  - "Releasing of Ctrl" after "Ctrl+Drag&Drop" interferes folder copy operation.

Comment 13

[WADA:World Anti-bad-Duping Agency]

> Folder structure in POP3 Account

Many lines for "/Inbox.sbd/Level 1" are seen, and size associated it is different. What size? What is difference? The list is generated by which command with which parameter?
> 2,8M /Inbox.sbd/Level 1
> 227M /Inbox.sbd/Level 1
>  40K /Inbox.sbd/Level 1
>  44K /Inbox.sbd/Level 1
>  14M /Inbox.sbd/Level 1
>  13M /Inbox.sbd/Level 1
> 2,0M /Inbox.sbd/Level 1
>  18M /Inbox.sbd/Level 1
>  10M /Inbox.sbd/Level 1

Comment 14

[paladin]

"Level n" represents a folder at level n. That is, If I have a folder named "Generic/Last Year" I have represented them as "Level 1/Level 2". In conclusion: Even though they are all named "Level 1" they are different folders!
Please look at the directory structure with this in mind.

The command used to generate the size report is "du -hS", which is run under a "for" cycle in bash for each folder in both the origin (POP3 account) and destination (Local Folders). The text files I attached contain the full directory structure of both origin and destination after copy process. I think it is obvious what the difference is. For example, there is no folder at level 4 in the destination! Also, there are some folders with different sizes:

5,1M	/Inbox.sbd/Level 1/Level 2/Level 3   |	5,0M	/Inbox.sbd/Level 1/Level 2/Level 3

I encourage you to run a "diff -y" of the two files and see the differences. With this command I think they will be obvious.

Regarding the size, please take into account that there are lots of folders that have the same size in the origin and destination. So I think we can rule out the "how I did the size report" and accept its values. If not, could you please explain me your theory?

I never heard of movemail before, so I think I'm not using it.

Finally, unfortunately during some maintenance I've remove the mail backup I was testing, so it will be hard to make further tests... :(

Comment 15

[Wayne Mery (:wsmwk)]

paladin 
Is it likely you got a new message or an alert from the time you started the drag to the failure?

Comment 16

[WADA:World Anti-bad-Duping Agency]

According to comment #15, bug opener can't produce his problem any more.
Wayne Mery, I think closing as INCOMPLETE is appropriate.

Comment 17

[Ludovic Hirlimann [:Usul]]

(In reply to comment #16)
> According to comment #15, bug opener can't produce his problem any more.
> Wayne Mery, I think closing as INCOMPLETE is appropriate.

You can do it too Wada :-)

Comment 18

[Wayne Mery (:wsmwk)]

given the detail provided by paladin I'd prefer this not be closed just yet
a) question for paladin in comment 15 is still important to me 
b) someone should attempt to reproduce
c) ? maybe getting bug out of General will get more eyes
d) if we do close, we might rather attempt to dupe to a bug with similar symptoms
reopening for the above

Comment 19

[paladin]

Sorry for not giving this enough attention and respective late reply.
I was using a copy of a user's mailbox on my computer. So, I had not account configured and it would not be possible to receive any new message.
Again, I'm sorry for not being able to test this further.

I do remembered something now that might be a clue: the original mail box was in a windows machine, but I was doing the tests under Linux. Would this be a reason for this error?

It is my (humble) opinion that you really should try to reproduce this bug. Or have I been the only one who had a similar problem?

Comment 20

[Wayne Mery (:wsmwk)]

Given the passage of timing and the fact that we don't have paladin's message store, I think the path of lowest effort with highest potential information gain is for paladin to try reproducing with version 3. If it fails with version 3, then we have something worth pursuing.

Comment 21

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to last question. If you feel this change was made in error, please respond to this bug with your reasons why.