Closed Bug 517198 Opened 12 years ago Closed 12 years ago

Compact doesn't expunge messages with IMAP deleted flag (CONDSTORE supported IMAP with "mark as deleted" model)

Categories

(Thunderbird :: Folder and Message Lists, defect)

defect
Not set
minor

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 3.0rc1

People

(Reporter: shopik, Assigned: Bienvenu)

References

Details

(Keywords: qawanted, regression, Whiteboard: [no l10n impact])

Attachments

(4 files)

I've noticed what menu compact or button compact sometimes does nothing. I've using mark as deleted model and compact suppose to expunge message when I use it. But instead it does some locall compact as I see progress bar appears and lots hard drive activity. Workaround is easy just restart TB
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090916 Shredder/3.0pre
most likely a duplicate of bug 462880
Not really and that not GMAIL depend bug
(In reply to comment #0)
> (snip) compact suppose to expunge message when I use it.
> But instead it does some local compact as I see progress bar appears and lots hard drive activity.

"comapction of offline-store by Compact" had been disabled by bug 420115(to avoid some problems. only EXPUNGE was executed after change by bug 420115), but it was already re-enabled by fix of bug 482476. (See Bug 499630 for brief history.)
Do you mean that EXPUNGE command was really not issued when "Compact"?
Nikolay Shopik, can you attach IMAP log for your problem to this bug?
> https://wiki.mozilla.org/MailNews:Logging
Attached file IMAP log
Here you go, startup TB, select message in bugzilla folder, delete it, press compact, it doesn't disappear from list of messages.
Appears like it expunging it but still showing it there.
Attachment #401686 - Attachment mime type: application/octet-stream → text/plain
Which delete model? (Server Settings, When I delete a message)
If other then "Just mark it as deleted", mail is to disappear upon "delete"...

Which folder view? "All Folders"? "Smart Folders"? Or other?
Single mail folder? Or saved search folder for multiple-folders/multiple-accounts?
I've mention this on comment #0 I'm using mark it as deleted. "All folders", no saved search folders at all. What I'm seeing additional it actually delete them just after I restart TB upon start.
Oh sorry, I missed "mark it as deleted" in comment #0.
When you deleted mails(or a mail finally), all mail in the folder was deleted?

Log says that a mail is deleted(store flag \Deleted) and expunge is issued. And, next flow occurs just after expunge.
> S-bugzilla:SendData: 17 UID fetch 5730:* (FLAGS)
> S-bugzilla:CreateNewLineFromSocket: * 2620 FETCH (UID 5729 FLAGS (\Seen NonJunk))
> S-bugzilla:CreateNewLineFromSocket: 17 OK Fetch completed.
It looks next situation.
  1. You additionally deleted a mail(last mail).
  2. You executed "Compact" then EXPUNGE was issued.
  3. Just after the EXPUNGE, Tb issued 17 UID fetch 5730:* (FLAGS)
  4. Tb knew that largest UID of existent mail=5729 by "* 2620 FETCH (UID 5729".  Tb fails to remove EXPUNGED mail data because of no mail of UID>=5730?

Copy a mail to folder of "bugzilla". Will deleted mails disappear?
If yes, it may be sibling of Bug500144/Bug 462880.
I didn't delete last mail, that was some mail in middle list. I've executed compact and tried to copy from INBOX to bugzilla folder, mail copied deleted mail still there. 
And as I said earlier if I restart TB mail will disappear during startup.
"LIST" after "SELECT" is required? Is next workaround?
(i)   Click other folder, wait for a while(MailDB close), click the folder again.
(ii)  Collapse IMAP account, wait for a while, re-expand account, click the folder again.
(iii) Click IMAP account, "Open" of context menu, click the IMAP folder at new Tb window. (close old Tb window).

Is "offline use" enabled for the folder? (Folder Properties/Synchronization)
Is "Gloda" enabled? (Tools/Options/Advanced, [ ? ] Enable Global Search Indexer)
Attached file working condition
Suddenly it stars working, but re-downloading all messages again in folder where I delete message. If I delete message and press compact it actually expunge it and list of messages are disappear it will show up if I select other folder and select back.
I'm sorry forget to answer your question. Yes I have all folder offline enabled and gloda enabled too
(In reply to comment #10)
> "LIST" after "SELECT" is required? Is next workaround?
> (i)   Click other folder, wait for a while(MailDB close), click the folder
> again.
> (ii)  Collapse IMAP account, wait for a while, re-expand account, click the
> folder again.
> (iii) Click IMAP account, "Open" of context menu, click the IMAP folder at new
> Tb window. (close old Tb window).
Got it reproduced again, none of these suggestions are help. Message are still there.
What is displayed at "Activity Manager"? "<account> is up to date" status?
Does rename of the folder work when problem exists?
If folder name at folder pane is not changed, click other folder and the folder(old name) again. Does similar phenomenon to Bug 513012 Comment #7 occur?
Nikolay Shopik(bug opener), please modify bug summary, in order to avoid misleading, for ease of understanding problem, for ease of search.
Summary: Compact sometimes doesn't work for expunge → Compact doesn't expunge messages with IMAP deleted flag
(In reply to comment #14)
> What is displayed at "Activity Manager"? "<account> is up to date" status?
Its up to date
> Does rename of the folder work when problem exists?
Not sure what are you talking about
> If folder name at folder pane is not changed, click other folder and the
> folder(old name) again. Does similar phenomenon to Bug 513012 Comment #7 occur?
No I don't see such problems I can remove folder and it hides immediately
(In reply to comment #16)
> > Does rename of the folder work when problem exists?
> Not sure what are you talking about

One of causes of Bug 513012 Comment #7 is "Mail DB is still opened by auto-sync(or indexer or something)" and it'll produce problem like that when rename too. AFAIK, "Compact" executes explicite mail folder open. It's question to know whether the "Mail DB is still opened by auto-sync(or indexer or something)" exists or not. 

In your case, EXPUNGE command is issued as expected. And, as written in comment #0, "compaction of local-store" looks to be invoked normally. Problem is that "marked as deleted mail(red cross, with strike line)" is not removed from thread pane. Your case is not "* 0 EXISTS" for LIST case.
My question is why "re-sync is not executed normally upon folder click again(re-open usually) in your case".
logs showing the behavior are in attachment 401666 [details] , this makes using fastmail unusable for me.
Flags: blocking-thunderbird3?
Whiteboard: [no l10n impact]
Without steps to reproduce, we can't block on this.  Even if we had steps to reproduce, I'm unsure whether we would block on this.  Can you describe why this makes fastmail unusable?

Also, does Thunderbird 2 have this problem?
Flags: blocking-thunderbird3? → blocking-thunderbird3-
Keywords: qawanted
(In reply to comment #19)
> Also, does Thunderbird 2 have this problem?

No it doesn't, this is regression and problem appears to be start few months ago
The issue here is that CONDSTORE doesn't work well with EXPUNGE and the imap delete model. Expunge returns sequence numbers, but we haven't built up the whole sequence array, since we're using CONDSTORE, and we don't know which messages have actually been expunged. I'll try to see how hard this will be to fix.
Assignee: nobody → bienvenu
Status: NEW → ASSIGNED
Attached patch proposed fixSplinter Review
If we're using the imap delete model, and have a folder with CONDSTORE turned on, such that only have a partial flag state, then after an expunge, we need to remove messages with the imap deleted flag set by hand from the db.

fastmail.fm is a good test case for this, since they have condstore turned on. Neil, if you can't test this as is, I could generate a test server build for someone to try and report back.
Attachment #406395 - Flags: superreview?(neil)
Attachment #406395 - Flags: review?(neil)
Well, if you give me a Linux binary build (preferably x86_64, but i686 works as well), then I would be glad to test this as well :).
Adding CONDSTORE to bug summary for ease of search.
Summary: Compact doesn't expunge messages with IMAP deleted flag → Compact doesn't expunge messages with IMAP deleted flag (CONDSTORE supported IMAP with "mark as deleted" model)
Linux build will be here in an hour - not sure what variety of linux it is, however. http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry
So what's the code path for an expunge with imapdelete model without condstore?
nsImapFlagAndUidState::ExpungeByIndex is called for each of the expunge indexes the server tells us about, and then FindKeysToDelete is called with a full uid state. There, we execute the code after this comment:

  // otherwise, we have a complete set of uid's and flags, so we delete
  // anything thats in existingKeys but not in the flag state, as well
  // as messages with the deleted flag set.

There's a possibility that ExpungeByIndex is doing the wrong with partial flag states, but that's a separate issue, and I can't quite convince myself that it is wrong...
(In reply to comment #21)
> The issue here is that CONDSTORE doesn't work well with EXPUNGE and the imap
> delete model. Expunge returns sequence numbers, but we haven't built up the
> whole sequence array, since we're using CONDSTORE, and we don't know which
> messages have actually been expunged. I'll try to see how hard this will be to
> fix.
Is this a normal EXPUNGE or a UID EXPUNGE or does it make no difference?
This is a normal expunge. A UID Expunge would allow us to pass the UID's expunged to the core code, instead of looking for messages with the deleted flag. However, UID Expunge is part of the UIDPLUS imap extension, so it's not quite standard. In practice, CONDSTORE servers probably all support UIDPLUS, so that would be an other approach, though I'd have to write code to issue the different protocol, and parse the different response, and pass it to the core backend.
OS: Windows XP → All
Hardware: x86 → All
Target Milestone: --- → Thunderbird 3.0rc1
pinging for r/sr - we're going into code freeze for TB 3.0 today
Attached file IMAP log
This is kind of weird. Trying with

Mozilla/5.0 (X11; U; Linux i686 (x86_64); cs; rv:1.9.1.5pre) Gecko/20091031 Shredder/3.0pre

and the problem seems to be back ... kind of, but even worse. Now when I expunged, the folder is empty, but all (even undeleted) messages are deleted as well. Unfortunately that makes my paid fastmail.fm account worse than useless for me (I have actually lost bunch of messages due to this), and I have to switch to GMail for time being.
Switching back request for blocking, because of data loss.
Flags: blocking-thunderbird3- → blocking-thunderbird3?
Whiteboard: [no l10n impact] → [no l10n impact][has patch, needs review neil]
Attachment #406395 - Flags: superreview?(neil)
Attachment #406395 - Flags: superreview?(bugzilla)
Attachment #406395 - Flags: review?(neil)
Attachment #406395 - Flags: review?(bugzilla)
Whiteboard: [no l10n impact][has patch, needs review neil] → [no l10n impact][has patch, needs review standard8]
not blocking though we will try to land this patch. This issue is strictly a view update issue, not the separate dataloss issue described (which I can't reproduce)
Flags: blocking-thunderbird3? → blocking-thunderbird3-
To everyone who worked this bug:

THANK YOU!!

This has plagued me and I was unable to find a reproducible test case. Your
work is greatly appreciated.

 - Saul
Attachment #406395 - Flags: superreview?(bugzilla)
Attachment #406395 - Flags: superreview+
Attachment #406395 - Flags: review?(bugzilla)
Attachment #406395 - Flags: review+
Attachment #406395 - Flags: approval-thunderbird3?
fixed on trunk.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Whiteboard: [no l10n impact][has patch, needs review standard8] → [no l10n impact]
Target Milestone: Thunderbird 3.0rc1 → Thunderbird 3.1a1
Attachment #406395 - Flags: approval-thunderbird3? → approval-thunderbird3+
Target Milestone: Thunderbird 3.1a1 → Thunderbird 3.0rc1
fixed on branch as well.
You need to log in before you can comment on or make changes to this bug.