571421 - Some deleted messages up show up in gloda searches, and os integrated search (Spotlight and Windows Search)

Status: NEW

(MailNews Core :: Database, defect)

Description

[David Ascher (:davida)]

I (and I think bwinton) have seen deleted messages show up in gloda.  I think I've figured out a first approximation at STRs, based on the messages I've seen.

Specifically, I seem to find messages in two folders (bugmail and amo-editors) where I follow a pattern of accumulating bunches of messages (tens, maybe hundreds), then doing a mass delete (ctrl-a + delete).

It strikes me as not unthinkable that I have at times quit TB or put the laptop to sleep and thus interrupt processing of subsequent events shortly after.  I don't think I've had recent crashes that would complicate things, so I don't think crashes are to blame.

It's also possible that I've done these mass deletes on my other TB installation, and hence that the delete notification is not making its way through the IMAP chain to my main laptop.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

_Very_ approximate STRs :)

The other TB installation case seems the most likely.  Gloda's handling of msgsDeleted notifications is very straightforward; we synchronously issue a SQL UPDATE to mark the messages deleted.  Assuming the SQL transaction gets committed, the messages should never be returned in search results after that point, regardless of whether deletion processing has been completed.

In the multiple installation case, the accusation would be that gloda is not receiving a msgsDeleted notification.  This could happen because of a mailnews bug or because gloda is not getting registered for notifications until after the IMAP code has found out about the change.

In the not-the-multiple-installation case, the most likely explanation would be that the SQL commit actually never ends up happening.  Either an automated rollback on the part of SQLite, a(n impossible?) rollback by the part of gloda code, or shutting down somehow without letting the commit run to completion could net out to be the same thing.

Since it sounds like both you and blake can generally reproduce this, I am going to wait on a more specific set of steps to reproduce, at least for this specific bug.

If you don't think it's likely that you will be able to provide more specifics, I have a shelved extension (idea) that sanity checks the current state of the gloda database against the folder databases to check gloda's correctness.  That would likely be useful in tracking down the problem without being a wild goose chase.

Comment 2

[David Ascher (:davida)]

I assume until your shelved extension gets unshelved, there's no way for me to clean up my database to be able to get to a clean state than scrapping the entire db, correct?

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Folder compaction cures all ills.  It ensures that there is an exact correspondence between messages that the message database and gloda think are in a folder.  Unfortunately, it's only a thing for local mail folders.

The proposed extension basically amounts to the existing folder compaction algorithm but more paranoid and it actually reports errors since someone probably cares about its output.

So, in the interim, marking an IMAP folder as "gloda, don't index that!" and then marking it as "gloda, go index that!" should correct things.  It just won't be very efficient as it amounts to marking everything in the folder deleted, then re-indexing all the messages in the folder, potentially un-deleting everything that still exists.

I am open to input on when to unshelve the extension, but either way would really benefit from more details about when this seems to happen since there are a lot of possibilities and without using the same IMAP server and operating system as you I may never see what you are seeing.

Comment 4

[David Ascher (:davida)]

The interim solution should help me see if the problem shows up again.

Comment 5

[David :Bienvenu]

Looking at the code, it's quite possible that we're not sending a msgDeleted notification to the folder notification service in the case of messages deleted because they're gone from the imap server. We still send the old-fashioned nsIFolderListener notification, but I believe gloda is (correctly) using the folder notification service for this notification.

This would also affect the os search integration (Spotlight and Windows Search)

Comment 6

[David :Bienvenu]

My mistake - we're sending a msgsDeleted notification in the case that we notice that an other connection has deleted a bunch of messages, when we select the folder. http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapMailFolder.cpp#2844