Open Bug 425735 Opened 16 years ago Updated 2 years ago

nested imap folder copy (via drag.n.drop) creates all subfolders, only copies mail in one

Categories

(MailNews Core :: Networking: IMAP, defect)

x86
Windows XP
defect

Tracking

(Not tracked)

UNCONFIRMED

People

(Reporter: pdp, Unassigned)

Details

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13
Build Identifier: version 2.0.0.12 (20080213)

have two imap mail accounts, am migrating mail from 1st to 2nd.  1st has a folder containing other folders containing messages.  Action is drag.n.drop copy of outer folder on 1st to 2nd.  Result is outer and inner folders created on 2nd, only mail from first inner folder is copied.  All other inner folders on 2nd are empty.

Reproducible: Always

Steps to Reproduce:
1.set up nested folders on 1st imap server, mail messages contained in inner folders
2.select outer folder on 1st account
3.drap.n.drop selected folder onto 2nd account
Actual Results:  
nested folders created on target imap server, only messages from first inner folder are copied.

Expected Results:  
copy all the messages in all of the nested folders

Scary bug, as a casual check of the first folder looks like the copy succeeded.  Paranoia saved me.
Get IMAP protocol log, and check real protocol level flow first.
See Bug 402793 Comment #1 for getting log.
If(and only if) Tb's fault is involved, attach log file via "Add an attachment" link (text/plain, if file size is accepted).
See RFC 3501 "6.4.7. COPY Command" for behavior/flow when COPY.
> http://www.faqs.org/rfcs/rfc3501.html  
Had you accessed the sub folders in question with thuderbird on that computer? 
Unclear.  I may have for a fraction of the sub-folders.  I certainly did not visit each.
I've just tried to copy a very large and deep folder tree between a POP account and the Local Folders by drag & drop and the process failed silencely. You can see the total size of each tree below:

  > du -sh domain.com/Inbox.sbd/
  2,4G	domain.com/Inbox.sbd/
  > du -sh Local\ Folders/temp.sbd/
  895M	Local Folders/temp.sbd/

I can confirm that this also happens when copying to an IMAP server.

I'm making these tests in Linux, using Thunderbird 2.0.0.19.
Just remembered, I'm running a AMD64 Debian Box.
I've just tried this on a 32bit Windows XP virtual machine and had the same result.
(In reply to comment #4)
> copy a very large and deep folder tree between a POP account and the Local Folders by drag & drop
>   > du -sh domain.com/Inbox.sbd/
>   2,4G    domain.com/Inbox.sbd/

AFAIK, "Drag&Drop of local mail folder between accounts" is not file copy of mail folder file for the mail folder.
Was "mail data with MSG_FLAG_EXPUNGED bit is ON in X-Mozilla-Status: header" copied to target mail folder file(folder of "Local Folders")?
See next web page for MSG_FLAG_EXPUNGED bit.
> http://www.eyrich-net.org/mozilla/X-Mozilla-Status.html?en
2.4GB even after "File/Compact Folders" of domain.com?
Well, Drag&Drop doesn't remove the mails from the originating account and, when the folder tree is small enough, it copies the mails to the destination folder. So, from my point of view it seems to be a copy and not a move. Anyway, I also tried Drag&Drop while pressing Ctrl and had the same results.

How should I check the MSG_FLAG_EXPUNGED? Should I do the binary math all the way through the 895MB of mail in the origin account and compare with the destination account?

Yes, that's the size even after "File/Compact Folders".
Is there any tool to check if MSG_FLAG_EXPUNGED is copied?
(In reply to comment #8)
> Is there any tool to check if MSG_FLAG_EXPUNGED is copied?

Test editor. Have you read "X-Mozilla-Status explained" by Christian Eyrich, which I pointed? Data in X-Mozilla-Status: is hexa-decimal notation of 2 bytes binary data. For example, "0009" is "0x0008 OR 0x0001". i.e. MSG_FLAG_EXPUNGED=On and MSG_FLAG_READ=On and all other bits is Off.

Note-1:
X-Mozilla-Status:/X-Mozilla-Status2: is written in local mail folder file only(file for local mail folder of POP3 accounts or pseudo account of "Local Folders").
Note-2:
Comparison of file size(file for local mail folder) has meaning only when "copy of local mail folder(local file) to local mail folder(local file)" and only when size of file for mail data(not ".msf").

To paladin@mail.pt:
I asked you about "size after compact folders" and "MSG_FLAG_EXPUNGED bit" because you said problem occurred even when "copy of local mail folders(POP3 account) to local mail folders(Local Folders)".
Show "Order Received" column(offset of mail data in file for mail folder. Note: when IMAP folder, it's UID of the mail). And check the value after "Copy folders then Comact Folders" or "Compact Folders then copy folders". Which mail in which folder hierarchy was not copied?
"Test editor" should be read "Text editor". Sorry for spam.
I want to do my best to help, but please understand that doing that check with a text editor for the 895MB of e-mails that were copied is an impossible task... when I asked for a tool I was hoping for an automated test tool. Is there any other way of doing this check? 

On Note-2:
Even so, I can use the result of mail size as a reference. I think you agree that a difference of 1.6GB is a bit too much... right?

I have this problem both while copying to Local Folders AND to IMAP. But I think that they may be related.
Mails fail to be copied after level 2, even though some are copied. But I still wasn't able to find a pattern, sorry. For example:

A
  B
    C
    D
  E
    F
    G
  H
  I

All folders will be created on destination, but contents of folders F to I will be empty.

Next Monday I will try to test the "Order received"'ich tests. Could you please try to explain them with a bit more detail? I didn't quite understood.

Many thanks.
(In reply to comment #11)
> doing that check with a text editor for the 895MB of e-mails

Single 895M file? Following is total size of files/directories under test.sbd, isn't it? Did you check size of each file used for mail folder?
> du -sh Local\ Folders/temp.sbd/
>  895M    Local Folders/temp.sbd/
(I read http://www.computerhope.com/unix/udu.htm for du command.)

To paladin@mail.pt:
Please distinguish at least next tree cases.
(1) IMAP to IMAP(bug opener's case).
    We are guessing that unknown mails in copy source folder(put in mail fodler
    by other client) is not copied((APPENDed) to target folder.
(2) Local to Local(your POP3 to "Local Folders" case).
    Only you says some mails are not copied.
    So I'm asking you to provide evidence.
(3) Local to IMAP(probably case you are trying to explain in your Comment #11).
    This case may be Drag&Drop issue. May be interruption/interference of
    Drag&Drap or IMAP connection. However, you said same as case (2). So I'm
    waiting for evidence of case (2).

Anyway, check case (2) first. Compare size of a file for folder "F"(of source account, POP3) and size of a file for folder "F"(of target account, "Local Folders"). Same size?
Check file content by text editor. Can you find difference?
Lets just start from the beginning.

Test 1:

Copy COMPLEX folder structure from a POP3 account to IMAP server. The total size of that complex folder structure is 2.4GB and each sub-folder can have from 1kB to 300MB of e-mail. The complex directory structure is created in the IMAP server and some of the e-mails are copied. Nevertheless, the copying process aborts silently. Many e-mails are missing in the destination folders and most folders are empty. I'm not able to understand what is the pattern of which e-mails are copied and which are missing.

Test 2:

Copy the same COMPLEX folder structure from a POP3 account to Local Folders. I obtain the same result as before, confirming that the problem is not in the copy to IMAP, but on the copy itself. This is what I started by reporting because I thought it would had something new to this bug report.

Generic:

Even though in the case I reported Thunderbird copied a total 895MB before silently aborting, there is no pattern in this. I've just tried again and it just copied something like 190MB.
The folder tree I showed you is an example! The folder tree I'm testing is much more complex.

Answers:

In the test I just did (which only copied 190MB), folder F is empty. Folders B an E were copied and are equal. Folder A is different. From the diff, some e-mails are missing between others that were copied. All other folders are empty.
Interruption of bulk copy by Drag&Drop?. AFAIR, there are some bugs for issue when "bulk move by Drag&Drop". In Local->IMAP case, "rejection by IMAP sever" may be involved(IMAP server sometimes has size limitation of upload in single operation)

To paladin@mail.pt:
As I wrote previously, this bug's original issue is different from yours. Can you open separate bug for "Local to Local(POP3 to "Local Folders") case?
(I think "Local to IMAP" is better to be processed after analysis of "Local to Local" case.)
FYI.
There are at least two kinds of issue.
(a) Bug 469455 : IMAP to IMAP. Unkown mail doesn't seem to be copied(appended)
> Bug 469455 Dragging "deep" trees from one IMAP account to another does not copy everything
(b) Bug 474468 opened by paladin@mail.pt. 
> Bug 474468 Drag&drop copy of e-mail between POP3 account to Local Folders aborts silently
I have a feeling they might be related.
One thing I DID find in my IMAP server's logs was related with the number of open files. But that seemed to be only related with the NFS I was writting on. Did anyone check the ulimit for open file descriptors in the server? (Talking about issue a) ;)
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
(In reply to comment #16)
> (Talking about issue a)

"addition of new mails" is notified by IMAP server to IMAP client via IDLE, and IDLE is per folder(per cached connection) mechanism, and Tb's default of "number of cached connection" is 5, and any IMAP server has limitation of it(setting such as MAXPERIP. Gmail IMAP' current limit = 10).

Tb seems to currently issue append(or uid copy) for already known mails only. To support copy of "mails added by other clients", re-synch of copy_source_folder is required, after copy_target_folder creation and before mail copy process. Please note: Even if the improvement will be made, hole still exists(new mail copied by other clients while Tb is copying can't be copied by Tb).
Gene, is it reasonable for us to support this?  

Even if it is reasonable, we should block this action until actually support it.  See also bug 589008
Flags: needinfo?(gds)
Wayne, I dragged the top level folder of a deeply nest folder tree on server A to a nested folder on server B and it seems to work fine. Good feature is that it does not try to actually "move" the folders as an intra-server folder drag does.

However, I did see one problem. If I delete the tree that I just copied onto server B, the folders and emails appear to be gone. However, the supposedly deleted container directories are still present, e.g., a-bear.sdb, but are empty. If I try to "re-drag" the tree to the same place, nothing happens. Deleting the extraneous *.sdb directories or "repairing" the folders does not help. I have to go into the profile and delete *all* the dirs and emails for the account under .../ImapMail/, and let all the folders and emails re-fetch from server so that a re-drag to the previous destination can occur again.

Re bug 589008: If you select two or more folders and right-click, there is no "delete" option. If you hit delete key, you get a warning about deleting *a* folder (not folders). If you go ahead and click OK, only the first folder in your selection is deleted. Also, if you drag multiply selected folders, only 1st one in the selection is actually moved or copied.
Flags: needinfo?(gds)
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.