541349 - (glodafailtracker) [META] [faceted search] incomplete/incorrect/inaccurate/missing messages in "Search all messages" results for Gloda Global Search and Indexer

Status: NEW

(Thunderbird :: Search, defect)

Description

[Wayne Mery (:wsmwk)]

+++ This bug was initially created as a clone of Bug 519202 -  incomplete/incorrect/inaccurate search or filter results [meta] +++

This bug is to focus on incomplete/incorrect search *results*, eg false positive or messages missing from results list for Gloda Global Search and Indexer.  message count and UI issues are specifically excluded. 

I haven't double checked every bug to make sure it belongs here.  Initial list:

bug 524954, bug 523183, bug 522222, bug 522908, bug 523173, bug 523183 

Please feel free to improve the query and update the bug list

Comment 1

[Thomas D. (:thomas8)]

An older tracking bug with a similar purpose, but probably of a more private, temporary, limited, and technical nature:

Bug 497199 - [META] tracking bug for regressions from Bug 474701, front end rework for new gloda global search ui

Comment 3

[Wayne Mery (:wsmwk)]

adding Bug 523572 - [faceted search] In "Show all as list" mode, then quick search "Message body filter" is broken (returns nothing) - although strictly speaking it's tangential

Comment 4

[Hero]

Attachment: Example of mail that is not indexed by Thunderbird.eml

This mail (from the inbox-folder) was not indexed by Thunderbird's search database. I even tried re-indexing everything (Close Thunderbird, delete global-messages-db.sqlite, open Thunderbird and waite...). It didn't help.

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to Hero from comment #4)
> Created attachment 8795305 [details]
> Example of mail that is not indexed by Thunderbird.eml
> 
> This mail (from the inbox-folder) was not indexed by Thunderbird's search
> database. I even tried re-indexing everything (Close Thunderbird, delete
> global-messages-db.sqlite, open Thunderbird and waite...). It didn't help.

In my testing, the message was first marked as a scam. And thus not indexed.
After telling thunderbird it was not a scam, it was marked as junk. And thus not indexed. 
After I marked the message as not junk it got indexed. And search was successful.

Comment 6

[Hero]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> (In reply to Hero from comment #4)
> > Created attachment 8795305 [details]
> > Example of mail that is not indexed by Thunderbird.eml
> > 
> > This mail (from the inbox-folder) was not indexed by Thunderbird's search
> > database. I even tried re-indexing everything (Close Thunderbird, delete
> > global-messages-db.sqlite, open Thunderbird and waite...). It didn't help.
> 
> In my testing, the message was first marked as a scam. And thus not indexed.
> After telling thunderbird it was not a scam, it was marked as junk. And thus
> not indexed. 
> After I marked the message as not junk it got indexed. And search was
> successful.

Hi!

Thank you for your response! 

I subscribe to varies newsletters which all get marked as scam by Thunderbird. This is one of the very few annoyances with Thunderbird. I always immediately tell Thunderbird that it is not a scam. But I don't know what you are referring to when you say it gets marked as junk. I don't see any such labelling in my Thunderbird. And the message is still not indexed. 

I look forward to hear from you again.

Best regards,
Henrik R.

Comment 7

[Hero]

(In reply to Hero from comment #6)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> > (In reply to Hero from comment #4)
> > > Created attachment 8795305 [details]
> > > Example of mail that is not indexed by Thunderbird.eml
> > > 
> > > This mail (from the inbox-folder) was not indexed by Thunderbird's search
> > > database. I even tried re-indexing everything (Close Thunderbird, delete
> > > global-messages-db.sqlite, open Thunderbird and waite...). It didn't help.
> > 
> > In my testing, the message was first marked as a scam. And thus not indexed.
> > After telling thunderbird it was not a scam, it was marked as junk. And thus
> > not indexed. 
> > After I marked the message as not junk it got indexed. And search was
> > successful.
> 
> Hi!
> 
> Thank you for your response! 
> 
> I subscribe to varies newsletters which all get marked as scam by
> Thunderbird. This is one of the very few annoyances with Thunderbird. I
> always immediately tell Thunderbird that it is not a scam. But I don't know
> what you are referring to when you say it gets marked as junk. I don't see
> any such labelling in my Thunderbird. And the message is still not indexed. 
> 
> I look forward to hear from you again.
> 
> Best regards,
> Henrik R.

The mentioned message is still not indexed by my Thunderbird 52.9.1 (64-bit). 
This type of error is very unfortunate. For me, it places TB under suspicion of censuring search results...

Comment 8

[Hero]

Attachment: This message, which I wrote as a reply, is also excluded from TB search database.

I signed this message with Enigmail PGP/GPG when I sent it.

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to Hero from comment #8)
> Created attachment 9011230 [details]
> This message, which I wrote as a reply, is also excluded from TB search
> database.
> 
> I signed this message with Enigmail PGP/GPG when I sent it.

If you signed AND encrypted then bug 188988

Comment 10

[Hero]

No, only signed.

Comment 11

[Hero]

Gloda (the global search database) excludes quotations.
In https://support.mozilla.org/en-US/questions/1239079#answer-1169021
- "Search: Message found with some keywords but not with others" - the conclusion is: 
"It would be helpful if there was an option to exclude, or not, quoted text from Global Search."
The problem is:
I often use 'Paste as Quotation' to insert something in a mail I am composing. So it would be VERY smart if TB could distinguish between quotation generated automatically by TB when you 'Reply' to a message, and blocks of text I insert as "quotations" in messages, and thus only exclude the quotations generated automatically by TB from gloda!

Comment 12

[Thomas D. (:thomas8)]

Adding papercut keyword so as to keep this on the radar. We should strive for some incremental improvements here. I'm aware that some of this will be non-trivial.