235004 - Compact | Folders (IMAP purge) updates inode change time (ctime) on *all* server's mail files, even if not changed

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Calum Mackay]

User-Agent:       
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7a) Gecko/20040218

Doing a Compact | Folders, remove email marked as deleted, causes the IMAP
server's mail file to have its UNIX inode change time (ctime) updated, even when
there were no changes made to the folder (i.e. no mails deleted).

Now that Compact | Folders actually compacts every single mail folder for this
account, this is suddenly a real problem: *all* my mail folders' ctimes are
updated, even for folders I've not viewed for months, let alone deleted emails from.

The reason this is a problem is backups: my nightly incrementals are now running
at 2GB, almost all of which are mail folders that have not been written to for
months. They're still backup up since their ctimes have changed.

e.g, these mail folders have not been written to (or even read) for almost a year:

diz $ ls -l accounting Cellini Feltech Honkers 
-rw-------    1 calum    calum       99926 2003-04-09 14:49 accounting
-rw-------    1 calum    calum        5483 2003-04-09 15:00 Cellini
-rw-------    1 calum    calum      240548 2003-04-09 15:06 Feltech
-rw-------    1 calum    calum       26510 2003-04-09 15:26 Honkers

yet they appear in my nightly incrementals, every night, since their ctimes get
updates every time I do a Compact | Folders:

diz $ ls -lc accounting Cellini Feltech Honkers 
-rw-------    1 calum    calum       99926 2004-02-20 11:44 accounting
-rw-------    1 calum    calum        5483 2004-02-20 11:44 Cellini
-rw-------    1 calum    calum      240548 2004-02-20 11:44 Feltech
-rw-------    1 calum    calum       26510 2004-02-20 11:44 Honkers

This causes my incrementals to fail, as the week progresses, since there's not
enough room on the tape :(

The IMAP server (Uow imapd 2002e, in my case) is what's actually doing this, of
course, but it's in response to mozilla's purge request.

Would it be possible not to do a folder purge if the folder contains no deleted
messages? I think this might be a workaround...

Reproducible: Always
Steps to Reproduce:
1.
2.
3.

Comment 1

[David :Bienvenu]

Yes, if the server told us there were no deleted messages, we could forego the
expunge. I'll try to get to this.

Comment 2

[Mikael Parknert]

Attachment: Possible patch

This works against my imap-server, but I'm not sure about the proper use of arg
'useUID' to Search

Comment 3

[David :Bienvenu]

Comment on attachment 161658 [details] [diff] [review]
Possible patch

Thx for looking into this. I think this should be controlled by a pref,
defaulted to false, since for most users/servers, this is just extra load on
the server, and will be slower on the client side as well.

Comment 4

[Mikael Parknert]

Attachment: patch with pref

Added a pref (only read on startup)

Comment 5

[[e-mail]]

Comment on attachment 161682 [details] [diff] [review]
patch with pref

Fair enough.

Comment 6

[David :Bienvenu]

Comment on attachment 161682 [details] [diff] [review]
patch with pref

I'll check this in (after fixing the braces style :-) ) We might ultimately
need to make this per-server since some servers don't support SEARCH at all,
but the group of people who care about this *and* run against multiple servers,
one of which doesn't support SEARCH, are probably vanishingly small.

Comment 7

[Calum Mackay]

what would the behaviour be on a server that didn't support search, with this
pref turned on?

I imagine that we would fail to expunge the folder, even if it had deleted mails?

Is there any way to tell whether a particular server has search functionality? I
assume if I can do a full body search, then it does? Or is that a different
"search".

oh. thanks very much for this, Mikael :)

Comment 8

[Mikael Parknert]

2004-10-12 10:17	bienvenu%nventure.com 	mozilla/ mailnews/ imap/ src/
nsImapProtocol.cpp 	1.578 	18/2  	
2004-10-12 10:17	bienvenu%nventure.com 	mozilla/ mailnews/ mailnews.js 	3.230 	5/2  

checked in by David

Comment 9

[Mikael Parknert]

(In reply to comment #7)
> what would the behaviour be on a server that didn't support search, with this
> pref turned on?

My guess is that the search command would fail, and then the EXPUNGE-command
would be sent.

Comment 10

[Calum Mackay]

i wasn't so sure; from the patch, I imagined that search() might return 0 if the
seach failed and then we would return without doing the expunge.

Comment 11

[Calum Mackay]

This doesn't seem to work, at least not for me. I tried both mozilla, and
thunderbird, builds, built locally from cvs this morning.

diz $ grep before_expunge $(locate nsImapProtocol.cpp)
    prefBranch->GetBoolPref("mail.imap.check_deleted_before_expunge",
&gCheckDeletedBeforeExpunge);

diz $ grep before_expunge ~/.*/default/*/*.js
/home/calum/.mozilla/default/pnvrv5p3.slt/prefs.js:user_pref("mail.imap.check_deleted_before_expunge",
true);
/home/calum/.thunderbird/default/rqmxjqe6.slt/prefs.js:user_pref("mail.imap.check_deleted_before_expunge",
true);
/home/calum/.thunderbird/default/rqmxjqe6.slt/user.js:user_pref("mail.imap.check_deleted_before_expunge",
true);

(restarting both apps, of course).

Compacting Folders, in either app, still resulted in mail files on the server
having their inode change time updated, even for files that haven't been changed
in months, and don't have any deleted messages:

diz $ ls -l pension
-rw-------  1 calum calum 546543 2004-01-22 01:25 pension
diz $ ls -lc pension
-rw-------  1 calum calum 546543 2004-10-15 00:47 pension

I'm using a UofW imapd server, running on localhost, over SSL.

Am I missing something obvious?

Comment 12

[Calum Mackay]

I also tested against a remote UoW imap server:

* OK [CAPABILITY IMAP4REV1 LITERAL+ SASL-IR LOGIN-REFERRALS STARTTLS AUTH=LOGIN]
xxx IMAP4rev1 2004.352 at Fri, 15 Oct 2004 01:03:32 +0100 (BST)

and the bug is still present there, too.

Comment 13

[Mikael Parknert]

(In reply to comment #12)
> I also tested against a remote UoW imap server:
> 
> * OK [CAPABILITY IMAP4REV1 LITERAL+ SASL-IR LOGIN-REFERRALS STARTTLS AUTH=LOGIN]
> xxx IMAP4rev1 2004.352 at Fri, 15 Oct 2004 01:03:32 +0100 (BST)
> 
> and the bug is still present there, too.

can you attach an imap-log?
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html

Comment 14

[Calum Mackay]

I've got an imap log but, since it's full of personal detail, would rather not
attach it here. Mikael: I could make it available to you personally, if you need
it? Alternatively, tell me what to look for, perhaps...

As a start, I will attach a snippet from the file, showing four mailboxes:

diz $ grep -i expunge thunderbird-debug.IMAP.snippet
1171262384[8d7dda8]:
8db4858:localhost:S-Mail/swan-sent/2003-02-partial.bz2:ProcessCurrentURL:imap://calum@localhost:993/Expunge%3E/notes_cdmnet:
 = currentUrl
1161489328[8c56f70]:
8cc4d08:localhost:S-Mail/swan-sent/2003-12.bz2:ProcessCurrentURL:imap://calum@localhost:993/Expunge%3E/Cellini:
 = currentUrl
1171262384[8d7dda8]:
8db4858:localhost:S-Mail/notes_cdmnet:ProcessCurrentURL:imap://calum@localhost:993/Expunge%3E/dave_barry:
 = currentUrl
1161489328[8c56f70]:
8cc4d08:localhost:S-Mail/Cellini:ProcessCurrentURL:imap://calum@localhost:993/Expunge%3E/WOTD:
 = currentUrl

none of which had deleted emails in. All four - in fact *every* mail folder -
had its UNIX file ctime updated during this operation.

Comment 15

[Calum Mackay]

Attachment: extract from IMAP protocol log

Attached snippet from the protocol log showing four mailboxes being expunged,
although they have no deleted messages.

Comment 16

[Mikael Parknert]

Comment on attachment 162302 [details]
extract from IMAP protocol log

8db4858:localhost:S-Mail/swan-sent/2003-02-partial.bz2:ProcessCurrentURL:imap:/
/calum@localhost:993/Expunge%3E/notes_cdmnet:  = currentUrl
>1171262384[8d7dda8]: 8db4858:localhost:S-Mail/swan-sent/2003-02-partial.bz2:SendData: 78 select "Mail/notes_cdmnet"
-----deleted 
>1171262384[8d7dda8]: 8db4858:localhost:S-Mail/notes_cdmnet:SendData: 79 SEARCH DELETED
>1171262384[8d7dda8]: ReadNextLine [stream=8d7e390 nb=10 needmore=0]
>1171262384[8d7dda8]: 8db4858:localhost:S-Mail/notes_cdmnet:CreateNewLineFromSocket: * SEARCH
>1171262384[8d7dda8]: ReadNextLine [stream=8d7e390 nb=24 needmore=0]
>1171262384[8d7dda8]: 8db4858:localhost:S-Mail/notes_cdmnet:CreateNewLineFromSocket: 79 OK SEARCH completed
>1171262384[8d7dda8]: 8db4858:localhost:S-Mail/notes_cdmnet:SendData: 80 UID fetch 1:* (FLAGS)


this shows that first we search for deleted (command # 79),
and since none is found, we do not send  80 EXPUNGE.

If you try this on a folder with messages to expunge, you
would have seen next command to be expunge.

Have you tried this in another imap-client?

Comment 17

[Mikael Parknert]

Attachment: imap log of expunge

this log shows two "Compact This Folder" - the first Inbox.Templates, the
second on Inbox. Templates don't need expunge, so none is sent, but Inbox needs
it.

Comment 18

[Calum Mackay]

Very interesting; so, we're not sending an expunge, and it's something else
we're doing that is causing the ctime to get updated.

I will try a different client...

Comment 19

[Mitch]

I can confirm this is NOT fixed. I just pulled cvs and rebuilt mozilla.
Expunging one mail folder, updates ctime on all folders of that server. And yes
i did have the appropriate config turned on in about:config

I'm using 
% telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LITERAL+ SASL-IR LOGIN-REFERRALS STARTTLS AUTH=LOGIN]
localhost IMAP4rev1 2004.352 at Sat, 16 Oct 2004 21:26:08 +0400 (GST)
100 logout
* BYE home IMAP4rev1 server terminating connection
100 OK LOGOUT completed

Comment 20

[Calum Mackay]

Not having any luck trying a different client. I have webmail, as an IMAP
client, but it doesn't have a purge all folders function, so I couldn't test that.

I'm beginning to think that this bug isn't really fixable in mozilla. I noted
that merely selecting a single mailbox immediately causes its ctime to be
updated. This is true of both mozilla and webmail.

So, if we're to avoid this, we mustn't select the mailbox; but then we don't
know whether it has any deleted msgs (and probably can't expunge it anyway).

So, I think the fix that Mikael kindly did doesn't actually fix the problem as
reported, which is that ctimes get updated unnecessarily, but that seems to be a
feature of the IMAP server, rather than anything we can control here.

So, the only fix for me is to turn off "compact all folders" and just do them
individually :(

thanks...