516232 - [faceted search] Deleted for good/expunged messages are shown in search results

Status: RESOLVED FIXED

(Thunderbird :: Search, defect)

Description

[Thomas D. (:thomas8)]

Messages that have been deleted for good (expunged) with <shift>+<del> continue to show up in search results, even after closing and reopening thunderbird.
They shouldn't, as soon as they are shift-deleted (while showing mails from trash is ok). Iow, when user shift-deletes a message, all relating data should be immediately removed from gloda, and never be re-indexed as long as the deleted-for-good-flag is set.

It's bad enough that shift-deleted messages physically remain in the folder file until you compress, but showing them in search results is definitely a considerable privacy issue --> major.

Comment 1

[Thomas D. (:thomas8)]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a1pre) Gecko/20090911 Shredder/3.1a1pre

Worse, even compacting does not make such messages go away, still shown in search results. That's a perfect failure in terms of privacy. Nominating blocking tb3.

Comment 2

[Thomas D. (:thomas8)]

Apart from compromising privacy, of course this will also render gloda-based searches completely unusable very quickly, as search results will continue to show false positives from the vast and (as per comment #1) ever-increasing choice of messages that have ever(!) been expunged. This should block, no doubt.

Comment 3

[Mark Banner (:standard8)]

Andrew, is this a duplicate of bug 465618?

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

No.  Fixing bug 465618 would make this problem less obvious, but the problem is GlodaMsgSearcher is building a query that can return messages that are marked deleted.  This happens because it is using hand-rolled SQL and has insufficient test coverage.

Comment 5

[Jens Müller (:tessarakt)]

(In reply to comment #0)
> They shouldn't, as soon as they are shift-deleted (while showing mails from
> trash is ok).

IMO, those should be grayed out, put at the end of the search results or something like that ...

Comment 6

[Thomas D. (:thomas8)]

Which are "those"? I assume you mean mails in trash? We definitely can't show expunged mails that have been shift-deleted.

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

Attachment: filter out deleted messages

This works, but it turns out our deletion processing might be more broken than I was thinking.  Using a local account, I deleted a message from my sent folder so it ended up in trash.  Then I deleted it from trash.  But it still showed up when deleted from trash.  Then I emptied the trash, and the search no longer found it.

Comment 8

[David :Bienvenu]

Comment on attachment 400558 [details] [diff] [review]
filter out deleted messages

this fixes my simple test of searching for messages I've just deleted from the imap trash.

Comment 9

[Thomas D. (:thomas8)]

(In reply to comment #7)
> Created an attachment (id=400558) [details]
> filter out deleted messages

Andrew, thanks for adressing this.
Which kind of deleted messages are you filtering out, trash-mails AND expunged or expunged-only?
I think user should have some way of including trash into the search. Either just include them, and/or gray them out, and/or put them at the end of search results (as per comment #5), and/or provide a checkbox to include them. After all, trash-mails still exist, and maybe the one you are looking for accidentally ended up in the bin.

> This works, but it turns out our deletion processing might be more broken than
> I was thinking. Using a local account, I deleted a message from my sent folder
> so it ended up in trash.  Then I deleted it from trash.  But it still showed
> up when deleted from trash.  Then I emptied the trash, and the search no
> longer found it.

This bug is not about the mails in trash, it's only about mails that have been expunged using *<shift>*-delete (in ANY folder, not just trash).
Do shift-deleted messages from other folders still show up after your patch (they shouldn't)?

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

pushed:
http://hg.mozilla.org/comm-central/rev/3cb3449994f6

Comment 11

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #9)
> Which kind of deleted messages are you filtering out, trash-mails AND expunged
> or expunged-only?

The filtering being performed is for correctness.  If a message is expunged, gloda should no longer know about it but for performance reasons we can't instantly perform all the purges (nor would we actually want to).  The filtering is of messages that are in limbo and technically should no longer exist.  Having said that, there are still bugs in deletion handling that are being worked on right now.  No guarantees until the unit tests land.

Gloda still knows about messages in the trash.