406127 - Imap folders copied to "Local Folders" account aren't readable below 1 level.

Status: RESOLVED WORKSFORME

(Thunderbird :: General, defect)

Description

[Glenn Meyer]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.10) Gecko/20071126 Ubuntu/7.10 (gutsy) Firefox/2.0.0.10
Build Identifier: Thunderbird 2.0.0.9 (20071031)

Copying a directory tree of about 100 mail directories and folders five levels deep from an imap server to the Local Folders directory of the client account produces a tree two levels deep, where only the top level folders' messages can be viewed. The next level folders are visible but appear to contain nothing. Folders below that level cannot be seen. I have tried copying the tree within the folders window of Thunderbird, and I have tried copying all the tree "directly" by creating a tarball on the server, ftp'ing that over to the client, and extracting the directories. In the former case, I get folders and directories one level deep. In the latter, I get a complete directory tree, with all the folders contents. But Thunderbird still can't see anything below level One. And yes, I have tried deleting all .msf files and letting Thunderbird rebuild them. Didn't help. 

Reproducible: Always

Steps to Reproduce:
1a. Drag and drop mail folders from imap server to "Local Folders" on client side.
or 
1b. Copy entire directory from imap server to "Local Folders" directory using your favorite UNIX/Linux utility.

NB: Mail directory should contain a few hundred directories and folders, to accurately reproduce my situation.  

2. Stop Thunderbird if you did 1a. 

3. Start Thunderbird. 

4. Attempt to browse "Local Folders" below the top level. 
Actual Results:  
Folder and contents of folders at the top level will be visible. Folders at second level will be visible, but will have no contents. No lower levels will be visible. 

Expected Results:  
Software should show entire n levels of mail directory, where n is the number of levels in the tree. For examples of programs that do this correctly, see pine, mutt, elm, evolution, and others that have been around for years or even decades. 

Fails the same on ubuntu 7.10, xubuntu 7.10, RHEL 4, RHEL 5, Centos5 and Centos 4.5.

Comment 1

[Magnus Melin [:mkmelin]]

Had you opened these lower level folders (so their headers were known)? I know when I did something similar, messages not yet "known" by that computer's thunderbird didn't get moved.

Comment 2

[Glenn Meyer]

Yes, I had. I'm assuming you mean opening the lower level folders before the move. Whether the folders had been opened or not, the lower level files' contents weren't visible, whether they were subdirectories or message folders.  

Comment 3

[Magnus Melin [:mkmelin]]

Can't reproduce with my test server, latest linux trunk build.

Comment 4

[WADA:World Anti-bad-Duping Agency]

> 1b. Copy entire directory from imap server to "Local Folders" directory using
your favorite UNIX/Linux utility.

To Glenn Meyer(bug opener):

Was your "Copy entire directory from imap server by utility" proper operation?

"mail folder hierarchy" is equal to directory hierarchy of file system at many IMAP servers.
But when local mail folder of Tb, "mail folder hierarchy" is implemented by   <folder>.sbd/<subfolder> structure of file system.
  <folder>     : file, no file extension, Unix mbox file
  <folder>.msf : file, mail summary file (probably Mork DB)
  <folder>.sbd : directory to hold files/directories for sub folders
    <folder>.sbd/<subfolder>      : file, no file extension. Unix mbox file
    <folder>.sbd/<subfolder>.msf  : file.
    <folder>.sbd/<subfolder>.sbd  : directory
      <folder>.sbd/<subfolder>.sbd/<sub_sub_folder>, ...
Check directory structure of "Local Folders" directory, which you created using utility, not created by Tb, first.

Because "Local Folders" is used for local copy of IMAP mail or import, test of "Copy entire directory from imap server by utility" is better to be executed to directory for dummy POP3 account.
 1. Define a dummy POP3 account, without "Global inbox" use.
    (assume server_name=dummy.pop3.server)
 2. Copy entire directory to dummy.pop3.server directory, and check structure
 3. After test, delete dummy account, and delete dummy.pop3.server directory

Comment 5

[Glenn Meyer]

I tried what you suggested, with the same result. The structure is not reproduced as above, but rather, as a normal UNIX file system. Perhaps what I need to do is put in a feature request in which Thunderbird, like evolution, pine, mutt, elm and others, is able to manage a "Local Folders" directory that is a UNIX directory. 

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #5)
> Thunderbird, (snip), is able to manage a "Local Folders" directory that is a UNIX directory.

"Local Folders" directory used by Tb is a normal/usual UNIX directory.
Simply, directory name used by Tb is different from your expectation.
You *HAVE TO* put a Unix mbox file(file without extension) under directory which is named <parent_folder_name>.sbd, if you want to use the Unix mbox file as "a mail sub folder of 'parent_folder_name' folder of Thunderbird" before your feature request will be implemented by you or other people.
Further, X-Mozilla-***: headers (***=Status/Status2/Keys) in local mail folder file, which are usually put by Tb automatically, are not generated by file copy operation. These headers are generated by Tb when "Compact Folder" (probably required twice). See Bug 377986 for this. 

Anyway, 1b. part is an ENHANCEMENT request, or INVALID bug report. 
Please focus on problem of 1a. (drag&drop of IMAP folder case).

Comment 7

[Glenn Meyer]

Attachment: Listing of subdirectory in Local Folders, after drag 'n' drop from imap server, from within Thunderbird.

This attachment shows the result of an attempt to copy a mailfolder, travel, from an imap server to the Thunderbird 2.0.0.9 "Local Folders" subdirectory. Note that after doing the copy, I did 10 Compact Folders over an hour's time, after which I closed Thunderbird, reopened it, re-examined the folder in Thunderbird, and, finally, closed Thunderbird and did at "ls -lR travel", replacing the userid and group id with blanks. Note that only four folders actually have any messages in them, and only at the top level.

Comment 8

[Glenn Meyer]

Please note that, in Comment #7, "copy" means "drag 'n' drop"

Comment 9

[Daryle Tilroe]

I can confirm this bug, there are also postings in the forums.

We are migrating to tbird version 2.0.0.14 (20080421) and switching IMAP servers at the same time.  To do this in a staged manner users are being setup with accounts on both servers and then cleaning/reorganizing/moving folder from one server to the other.  When copying (ie. dragging) a folder with subfolders from one IMAP server to another any subfolders that have not been accessed end up empty.  It seems likely that tbird relies on the header cache rather than actually looking into the contents of the source server.  This probably means it might be missing messages that were copied into folders by some other client if it has never refreshed its cache.  Thus even outside of our unusual situation of a mass, combined, migration users might be loosing emails if they copy folders which have been populated by some other imap client other than the one doing the copy (since its cache is not up to date).

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Linux i686;
> Build Identifier: Thunderbird 2.0.0.9 (20071031)
> 1b. Copy entire directory from imap server to "Local Folders" directory using your favorite UNIX/Linux utility.

FYI.
Your request of "1b" was already implemented "movemail support".
> http://www.mozilla.org/mailnews/movemail/index.html

Comment 11

[Magnus Melin [:mkmelin]]

Glenn: are you able to test with an alpha? (Or nightly build)
http://www.mozillamessaging.com/en-US/thunderbird/early_releases/

Comment 12

[Daryle Tilroe]

Just out of curiosity is the consensus that this bug won't be fixed in Tbird 2 but only in 3?  Also someone with the rights might want to consider changing the status to "NEW" and perhaps the title to a more generic description of the issue, eg: "Messages in never opened and uncached IMAP sub-folders are not copied to another account when parent folder is."

Comment 13

[Glenn Meyer]

I wouldn't do that even if I could. I occasionally recheck this "feature" of thunderbird, and find that even recently accessed subfolders (as in, subfolders accesssed within the last five minutes) don't copy. 

Comment 14

[Wayne Mery (:wsmwk)]

It will help greatly helpful if we know you can (or can't) reproduce this using the current beta as suggested in comment 11. We are now at beta 2. Just backup your profile before using
 http://www.mozillamessaging.com/en-US/thunderbird/early_releases/

No. This will not get fixed in Thunderbird 2.

Comment 15

[Wayne Mery (:wsmwk)]

I believe this is WFM on trunk, so closing WFM. If you can reproduce using trunk, beta (3 was just released) or newer version please reopen the bug. Thanks.