298033 - Using #mh namespace, all message headers are re-downloaded every session

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Bryan Wright]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040507
Build Identifier: Thunderbird 1.0.2-1.3.3 (20050513)

At least with #mh/ namespaces, Thunderbird seems to want to download all message
headers for a given folder once during each session.  This can take several
minutes for a large folder.  After the headers are downloaded, things are snappy
the next time I enter that folder during the same session.  But if I close
Thunderbird and re-open it, all of the headers are downloaded again the first
time I click on the folder.

This doesn't happen when I use another mail client (pine) with the same server
and the same folders. (This is using pine's imap support, and the #mh/
namespace, as with Thunderbird.)

Reproducible: Always

Steps to Reproduce:
1. Point Thunderbird at a #mh/ folder containing several thousand messages
2. wait.......wait.....wait.....
3. Browse through mail.
4. Close Thunderbird.
5. Start Thunderbird again.
6. Point Thunderbird at a #mh/ folder containing several thousand messages
7. wait.......wait...wait....
8. Now repeat this for each of the dozens of folders you visit during the 
   course of a day.
9. Decide not to use Thunderbird.

Actual Results:  
There are long pauses the first time each folder is accessed during a given session.

Expected Results:  
Quickly display a list of messages in the folder, fetching only headers for new
messages.

Comment 1

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 2

[Bryan Wright]

Hi folks,

The bug persists in version 1.0.6-1.1.fc3 (20050720), installed on FC2 from
thunderbird-1.0.6-1.1.fc3.i386.rpm.

My guess is that nobody cares enough about the #mh namespace to look into it.

Comment 3

[David :Bienvenu]

have you tried a 1.5 beta build? If I had access to a server that tbird was
having this problem with, I'd be happy to look into it.

Comment 4

[Bryan Wright]

I just tried version 1.5b1, and it has the same behavior.  Specifically:

Time to display an index of a #mh mailbox containing 5316 messages:

  Thunderbird 1.5b1: 4 minutes, 43 seconds (spent "Downloading message headers")
  pine:              27 seconds

Comment 5

[Wayne Mery (:wsmwk)]

Bryan, do you still see this problem.

Comment 6

[bryan-ff]

Hi Wayne,
I don't have any way to test it currently.  I'm no longer running an
imap server on the machine where I have my MH mail.  I'm still using 
exmh as a mail client.

Comment 7

[Wayne Mery (:wsmwk)]

Macmel, ludo, is this anything you can test?

Comment 8

[Ludovic Hirlimann [:Usul]]

(In reply to comment #7)
> Macmel, ludo, is this anything you can test?

We could ask gozer to configure a server with mh mailboxes and test against that. However from http://www.washington.edu/imap/documentation/formats.txt.html :

. mh	This is supported for compatibility with the past.  This is
	the format used by the old mh program.

	mh is very inefficient; the entire directory must be read
	and each file stat()'d, and in order to determine the size
	of a message, the entire file must be read and newline
	conversion performed.

	mh is deficient in that it does not support any permanent
	flags or keywords; and has no means to store UIDs (because
	the mh "compress" command renames all the files, that's
	why)

The later might be the cause of TB's re-download every time. I suppose Bryan was using an imap server when he reported the issue. If not then I would be very interested in what the #mh namespace is.

Comment 9

[Nikolay Shopik]

There is know issue with "#" symbol on trunk, I've reported it and it was fixed for beta 1 or 2.

Comment 10

[Wayne Mery (:wsmwk)]

bienvenu, given comment 8, do we care about this enough to even accept a patch if offered?  

(In reply to Nikolay Shopik from comment #9)
> There is know issue with "#" symbol on trunk, I've reported it and it was
> fixed for beta 1 or 2.

Bug 434850 which duped to bug 115091

Comment 11

[David :Bienvenu]

(In reply to Wayne Mery (:wsmwk) from comment #10)
> bienvenu, given comment 8, do we care about this enough to even accept a
> patch if offered?  
no, as long as the # is fixed.

Comment 12

[WADA:World Anti-bad-Duping Agency]

If IMAP folder of "# in folder name" exists, issue of bug 720911 bug occurs.
(no namespace, delimiter ="/", no "IMAP server directory:" setting)
Reason is perhaps  following;
     Unescaped folder name with #  !=  file name for the folder name with #
  && Escaped   folder name with #  !=  file name for the folder name with #
  Note: file name is hashed in Tb if "#" is contained in folder name,
        in order to avoid problems caused by "# is delimiter of hash in URL".
Cause of many problems in bug 720911 is that the "Unescaped folder name with #" is put in <servername>.msf file for <servername> DIRECTORY under ImapMail as "actual folder name" of "Tb's FOLDER named <servername>", as done on folder name with illegal file name characters. Due to this, root folder is changed from "/" to "/folder name with #" in Tb.

Phenomenon of this bug in recent Tb may be different from original comment #0, because code around "# in folder name" was changed to resolve # related problems.
Phenomena in recent Tb when "namespace=#mh/" and/or "namespace=#mh." may be different from phenomena observed in bug 720911. However, I believe that similar problems to bug 720911 surely occur in "namespace=#mh/" case too.

Setting dependency to bug 720911.

Comment 13

[WADA:World Anti-bad-Duping Agency]

Bug 697789 is perhaps a phenomenon in recent Tb when "#xxx/" or "#xxx." is used as namespace.

Comment 14

[WADA:World Anti-bad-Duping Agency]

As for folder access to processing mail(list, lsub, select, fetch, copy, store flag, ...), funny IMAP log was not seen in quick test for Bug 697789.

However, following files were created during test.
  8971b6d4.msf   (folder=#XYZ#) initially created by "create #XYZ#".
  suffix is incremented by internal unsubscribe/subscribe after initial creation.
  8971b6d4-1.msf (folder=#XYZ#)
  8971b6d4-1.sbd 
  8971b6d4-1.sbd￥c076ed7c.msf (folder=#XYZ#/#PQR#)
  8971b6d4-1.sbd￥c076ed7c.sbd

If something wrong occurs around 8971b6d4-1.msf(e.g. outdated msf condition -> deletion of 8971b6d4-1.msf) and if relation between #XYZ# and current 8971b6d4-1.msf/sbd is lost, any file/directory in 8971b6d4-1.sbd is lost, because no one knows about 8971b6d4-1.sbd any more. In this case, all mail data of subfolders under #XYZ#, #XYZ#/#PQR# is re-downloaded.
If normal combination of 8971b6d4.msf <-> #XYZ# and 8971b6d4.sbd￥c076ed7c.msf/sbd <-> #XYZ#/#PQR#, deletion of 8971b6d4.msf doesn't cause loss of subfolder's data in .sbd directory, because 8971b6d4.msf is re-created upon server access and 8971b6d4.sbd￥c076ed7c.msf/sbd can normally be accessed.

Because bug opener says like next, 
> 9. Decide not to use Thunderbird.
"Outdatd msf condition during following restart ob Tb" is suspected.
"Suffix in msf file name when #mh exist" may be cause of re-download of mails in this bug's case.

Comment 15

[WADA:World Anti-bad-Duping Agency]

As I wrote in bug 697789 comment #10, I could observe following IMAP log for "# and ^ in folder name" with Tb 9.0.1, with Gmail IMAP(# is allowed but ^ is not allowed in Gmail Label), by forcing namespace="#XYZ#/" after creation of #XYZ# folder.  
> :imap.gmail.com:S-INBOX:ProcessCurrentURL:imap://[e-mail]:993/create%3E/%23XYZ%23%5ESent/AAA:  = currentUrl
> :imap.gmail.com:S-INBOX:SendData: 9 create "#XYZ#/#XYZ#^Sent/AAA" 
> :imap.gmail.com:S-INBOX:CreateNewLineFromSocket: 9 NO [CANNOT] Folder name is not allowed. (Failure)
And following data in ...\ImapMail\imap.gmail.com.msf was observed, as observed in bug 720911.
> <(80=1)(82=#XYZ#^Sent)(87=40404400)>

I never did send operation nor "copy to Sent folder" operation. As for Sent folder, I merely opened Copies&Folders setting panel. This log indicates that Tb wrongly generates internal path for "Sent" from mail.identity.idX.fcc_folder = imap://[e-mail]/Sent.
This "#XYZ#^Sent" phenomenon looks "namespace=#XYZ/" or "namespace=#XYZ." only issue.
"/" in "#XYZ#/Sent" is escaped by "^"  because of string in hash of URL? 

When "namespace with #" case, this kind of issue may cause folder access failure which forces re-download of mails.

Comment 16

[WADA:World Anti-bad-Duping Agency]

FYI.
If non-personal namespace(shared/public namespace, other users namespace) is used, and if folder(or parent folder) for non-personal namespace is accessed and is shown at folder pane of Tb(returned as LSUB command and/or LIST command), Bug 540378 currently occurs.

Comment 17

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.