517198 - Compact doesn't expunge messages with IMAP deleted flag (CONDSTORE supported IMAP with "mark as deleted" model)

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[Nikolay Shopik]

I've noticed what menu compact or button compact sometimes does nothing. I've using mark as deleted model and compact suppose to expunge message when I use it. But instead it does some locall compact as I see progress bar appears and lots hard drive activity. Workaround is easy just restart TB
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090916 Shredder/3.0pre

Comment 1

[Matěj Cepl]

most likely a duplicate of bug 462880

Comment 2

[Nikolay Shopik]

Not really and that not GMAIL depend bug

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> (snip) compact suppose to expunge message when I use it.
> But instead it does some local compact as I see progress bar appears and lots hard drive activity.

"comapction of offline-store by Compact" had been disabled by bug 420115(to avoid some problems. only EXPUNGE was executed after change by bug 420115), but it was already re-enabled by fix of bug 482476. (See Bug 499630 for brief history.)
Do you mean that EXPUNGE command was really not issued when "Compact"?
Nikolay Shopik, can you attach IMAP log for your problem to this bug?
> https://wiki.mozilla.org/MailNews:Logging

Comment 4

[Nikolay Shopik]

Attachment: IMAP log

Here you go, startup TB, select message in bugzilla folder, delete it, press compact, it doesn't disappear from list of messages.

Comment 5

[Nikolay Shopik]

Appears like it expunging it but still showing it there.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Which delete model? (Server Settings, When I delete a message)
If other then "Just mark it as deleted", mail is to disappear upon "delete"...

Which folder view? "All Folders"? "Smart Folders"? Or other?
Single mail folder? Or saved search folder for multiple-folders/multiple-accounts?

Comment 7

[Nikolay Shopik]

I've mention this on comment #0 I'm using mark it as deleted. "All folders", no saved search folders at all. What I'm seeing additional it actually delete them just after I restart TB upon start.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Oh sorry, I missed "mark it as deleted" in comment #0.
When you deleted mails(or a mail finally), all mail in the folder was deleted?

Log says that a mail is deleted(store flag \Deleted) and expunge is issued. And, next flow occurs just after expunge.
> S-bugzilla:SendData: 17 UID fetch 5730:* (FLAGS)
> S-bugzilla:CreateNewLineFromSocket: * 2620 FETCH (UID 5729 FLAGS (\Seen NonJunk))
> S-bugzilla:CreateNewLineFromSocket: 17 OK Fetch completed.
It looks next situation.
  1. You additionally deleted a mail(last mail).
  2. You executed "Compact" then EXPUNGE was issued.
  3. Just after the EXPUNGE, Tb issued 17 UID fetch 5730:* (FLAGS)
  4. Tb knew that largest UID of existent mail=5729 by "* 2620 FETCH (UID 5729".  Tb fails to remove EXPUNGED mail data because of no mail of UID>=5730?

Copy a mail to folder of "bugzilla". Will deleted mails disappear?
If yes, it may be sibling of Bug500144/Bug 462880.

Comment 9

[Nikolay Shopik]

I didn't delete last mail, that was some mail in middle list. I've executed compact and tried to copy from INBOX to bugzilla folder, mail copied deleted mail still there. 
And as I said earlier if I restart TB mail will disappear during startup.

Comment 10

[WADA:World Anti-bad-Duping Agency]

"LIST" after "SELECT" is required? Is next workaround?
(i)   Click other folder, wait for a while(MailDB close), click the folder again.
(ii)  Collapse IMAP account, wait for a while, re-expand account, click the folder again.
(iii) Click IMAP account, "Open" of context menu, click the IMAP folder at new Tb window. (close old Tb window).

Is "offline use" enabled for the folder? (Folder Properties/Synchronization)
Is "Gloda" enabled? (Tools/Options/Advanced, [ ? ] Enable Global Search Indexer)

Comment 11

[Nikolay Shopik]

Attachment: working condition

Suddenly it stars working, but re-downloading all messages again in folder where I delete message. If I delete message and press compact it actually expunge it and list of messages are disappear it will show up if I select other folder and select back.

Comment 12

[Nikolay Shopik]

I'm sorry forget to answer your question. Yes I have all folder offline enabled and gloda enabled too

Comment 13

[Nikolay Shopik]

(In reply to comment #10)
> "LIST" after "SELECT" is required? Is next workaround?
> (i)   Click other folder, wait for a while(MailDB close), click the folder
> again.
> (ii)  Collapse IMAP account, wait for a while, re-expand account, click the
> folder again.
> (iii) Click IMAP account, "Open" of context menu, click the IMAP folder at new
> Tb window. (close old Tb window).
Got it reproduced again, none of these suggestions are help. Message are still there.

Comment 14

[WADA:World Anti-bad-Duping Agency]

What is displayed at "Activity Manager"? "<account> is up to date" status?
Does rename of the folder work when problem exists?
If folder name at folder pane is not changed, click other folder and the folder(old name) again. Does similar phenomenon to Bug 513012 Comment #7 occur?

Comment 15

[WADA:World Anti-bad-Duping Agency]

Nikolay Shopik(bug opener), please modify bug summary, in order to avoid misleading, for ease of understanding problem, for ease of search.

Comment 16

[Nikolay Shopik]

(In reply to comment #14)
> What is displayed at "Activity Manager"? "<account> is up to date" status?
Its up to date
> Does rename of the folder work when problem exists?
Not sure what are you talking about
> If folder name at folder pane is not changed, click other folder and the
> folder(old name) again. Does similar phenomenon to Bug 513012 Comment #7 occur?
No I don't see such problems I can remove folder and it hides immediately

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #16)
> > Does rename of the folder work when problem exists?
> Not sure what are you talking about

One of causes of Bug 513012 Comment #7 is "Mail DB is still opened by auto-sync(or indexer or something)" and it'll produce problem like that when rename too. AFAIK, "Compact" executes explicite mail folder open. It's question to know whether the "Mail DB is still opened by auto-sync(or indexer or something)" exists or not. 

In your case, EXPUNGE command is issued as expected. And, as written in comment #0, "compaction of local-store" looks to be invoked normally. Problem is that "marked as deleted mail(red cross, with strike line)" is not removed from thread pane. Your case is not "* 0 EXISTS" for LIST case.
My question is why "re-sync is not executed normally upon folder click again(re-open usually) in your case".

Comment 18

[Matěj Cepl]

logs showing the behavior are in attachment 401666 [details] , this makes using fastmail unusable for me.

Comment 19

[Dan Mosedale (:dmosedale, :dmose)]

Without steps to reproduce, we can't block on this.  Even if we had steps to reproduce, I'm unsure whether we would block on this.  Can you describe why this makes fastmail unusable?

Also, does Thunderbird 2 have this problem?

Comment 20

[Nikolay Shopik]

(In reply to comment #19)
> Also, does Thunderbird 2 have this problem?

No it doesn't, this is regression and problem appears to be start few months ago

Comment 21

[David :Bienvenu]

The issue here is that CONDSTORE doesn't work well with EXPUNGE and the imap delete model. Expunge returns sequence numbers, but we haven't built up the whole sequence array, since we're using CONDSTORE, and we don't know which messages have actually been expunged. I'll try to see how hard this will be to fix.

Comment 22

[David :Bienvenu]

Attachment: proposed fix

If we're using the imap delete model, and have a folder with CONDSTORE turned on, such that only have a partial flag state, then after an expunge, we need to remove messages with the imap deleted flag set by hand from the db.

fastmail.fm is a good test case for this, since they have condstore turned on. Neil, if you can't test this as is, I could generate a test server build for someone to try and report back.

Comment 23

[Matěj Cepl]

Well, if you give me a Linux binary build (preferably x86_64, but i686 works as well), then I would be glad to test this as well :).

Comment 24

[WADA:World Anti-bad-Duping Agency]

Adding CONDSTORE to bug summary for ease of search.

Comment 25

[David :Bienvenu]

Linux build will be here in an hour - not sure what variety of linux it is, however. http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry

Comment 26

[David :Bienvenu]

linux build with the fix:

http://s3.mozillamessaging.com/build/try-server/2009-10-15_10:[e-mail]-1255628912/[e-mail]2

Comment 27

[[e-mail]]

So what's the code path for an expunge with imapdelete model without condstore?

Comment 28

[David :Bienvenu]

nsImapFlagAndUidState::ExpungeByIndex is called for each of the expunge indexes the server tells us about, and then FindKeysToDelete is called with a full uid state. There, we execute the code after this comment:

  // otherwise, we have a complete set of uid's and flags, so we delete
  // anything thats in existingKeys but not in the flag state, as well
  // as messages with the deleted flag set.

There's a possibility that ExpungeByIndex is doing the wrong with partial flag states, but that's a separate issue, and I can't quite convince myself that it is wrong...

Comment 29

[Matěj Cepl]

(In reply to comment #26)
> linux build with the fix:
> 
> http://s3.mozillamessaging.com/build/try-server/2009-10-15_10:[e-mail]-1255628912/[e-mail]2

Yes, this actually works. Thank you.

Comment 30

[[e-mail]]

(In reply to comment #21)
> The issue here is that CONDSTORE doesn't work well with EXPUNGE and the imap
> delete model. Expunge returns sequence numbers, but we haven't built up the
> whole sequence array, since we're using CONDSTORE, and we don't know which
> messages have actually been expunged. I'll try to see how hard this will be to
> fix.
Is this a normal EXPUNGE or a UID EXPUNGE or does it make no difference?

Comment 31

[David :Bienvenu]

This is a normal expunge. A UID Expunge would allow us to pass the UID's expunged to the core code, instead of looking for messages with the deleted flag. However, UID Expunge is part of the UIDPLUS imap extension, so it's not quite standard. In practice, CONDSTORE servers probably all support UIDPLUS, so that would be an other approach, though I'd have to write code to issue the different protocol, and parse the different response, and pass it to the core backend.

Comment 32

[David :Bienvenu]

pinging for r/sr - we're going into code freeze for TB 3.0 today

Comment 33

[Matěj Cepl]

Attachment: IMAP log

This is kind of weird. Trying with

Mozilla/5.0 (X11; U; Linux i686 (x86_64); cs; rv:1.9.1.5pre) Gecko/20091031 Shredder/3.0pre

and the problem seems to be back ... kind of, but even worse. Now when I expunged, the folder is empty, but all (even undeleted) messages are deleted as well. Unfortunately that makes my paid fastmail.fm account worse than useless for me (I have actually lost bunch of messages due to this), and I have to switch to GMail for time being.

Comment 34

[Matěj Cepl]

Switching back request for blocking, because of data loss.

Comment 35

[David :Bienvenu]

not blocking though we will try to land this patch. This issue is strictly a view update issue, not the separate dataloss issue described (which I can't reproduce)

Comment 36

[Saul Tannenbaum]

To everyone who worked this bug:

THANK YOU!!

This has plagued me and I was unable to find a reproducible test case. Your
work is greatly appreciated.

 - Saul

Comment 37

[David :Bienvenu]

fixed on trunk.

Comment 38

[David :Bienvenu]

fixed on branch as well.