775024 - quick search and filters based on message body give wrong results

Status: NEW

(Thunderbird :: Search, defect)

Description

[Luis Menina]

User Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1
Build ID: 20120614114901

Steps to reproduce:

My test emails were automated emails sent from a Mantis bugtracker (mantisbt.org). They all had as the second line of the body, the following text:
"A NOTE has been added to this issue. "

1. enable the quick search, relying only on the email body.
2. enter a word you know is in the message body of several emails (here: NOTE)

The emails are simple emails (ie. non-multipart), encoded in base64


Actual results:

In my case, only 1 out of 3 emails was in the search result


Expected results:

All the emails with the searched term should have been in the search result

Comment 1

[Luis Menina]

This problem also appears in the filters relying on the message body. The body is the only thing I can use to determine if a Mantis bug is assigned to me. I could find no workaround, so I now have to manually go through a huge pile of emails every day just to find out if some action from me is required.

That's how I saw that filters were broken, and then, that quick search was broken.

I updated Thunderbird this morning, and this bug still happens in Thunderbird 14.

FYI, normal search is also broken, but in a slightly different way: it gives me in results emails that are not in the folder searched. I get 5 results with only 3 emails in the folder. When clicking on the 2 that shouldn't be there, I just get a blank window.

Comment 2

[:aceman]

Can you export those 3 messages as EML and attach them here?

Comment 3

[Luis Menina]

I though of it, but as this is professional material, I'm not sure I can. I'll try however to reproduce that with some harmless email, bug maybe in your own emails you'd be able to reproduce this bug. BTW, i'm not sure it is important, but I'm using an IMAP email account.

Comment 4

[Shari Smith]

Not sure if my issue is the same..
Windows 7 Thunderbird 14
Messages in a IMAP folder are not found via Quick Filter, however if I copy/move them to a local folder they are.

I have the same setup on a different PC using Earlybird 16.0a2 and the message in the IMAP folder is found.

Comment 5

[Shari Smith]

This is still broke in 15, but is still working in 16.

Comment 6

[Shari Smith]

I just installed Earlybird (16.0a2) and it's not working on my laptop although the quick filter works on my desktop. I can find now reason it would work on one system and not the other. The setup is exactly the same on both.

Comment 7

[Shari Smith]

I created a new profile, and this fixed the issue for 15 & version 16. Not reinstalling 14, so don't know if that was the issue for that as well. So it must have something to do with a carryover from an older version.

Comment 8

[Luis Menina]

I since then upgraded to Thunderbird 14 and 15. I created a new profile and checked with Thunderbird 15: this is still broken. I can reproduce it easily with Mantis bugmail, on different folders.

This seems heavily related to the fact that Mantis bugmail is Base64-encoded.

I thought that could be related to the date of the email, but on one folder it fails on mails of the day, and on another folder it fails on older mails.

Comment 9

[Luis Menina]

I've tried with other base-64 encoded messages I have in another mail account, with a clean profile, and the search works. So maybe the bug only triggers with Mantis bugmail...

Comment 10

[Luis Menina]

I'm running Thunderbird + Lightning in French on Debian. I'm using the binaries provided by Mozilla, not the Debian ones.

I've checked with some work colleagues:
- Colleague A uses Thunderbird 3.1.20 + Ligthning 1.02b in French on an old Ubuntu. He has the same bug
- Colleague B uses Thunderbird 11.0.1 + Lightning doesn't in English on Ubuntu. He doesn't have the bug. He's using the Thunderbird shipped by Ubuntu, without the Thunderbird localization packages, so it is in english.

Comment 11

[Luis Menina]

Ok, in fact colleague B has the bug too. On a folder with 1100+ emails of that kind, a search on a word present in all of them only shows 312 results.

Comment 12

[Luis Menina]

Attachment: mail sample 1

Comment 13

[Luis Menina]

Attachment: mail sample 2

Comment 14

[Luis Menina]

Attachment: mail sample 3

Comment 15

[Luis Menina]

For example, search fails to find anything in mail sample one. Even a simple a-z letter. This confirms what I saw one my B colleague machine, which always shows 312 messages for searches that should return the whole set of messages in that folder. So we think that may be related to the way the search index or database or whatever is constructed. If that assumption is right, you won't see the problem by just seeking in the sample emails using Thunderbird's search routines...

Please give me some advice on how I could help you reproduce the bug.

Comment 16

[:aceman]

I do not see the problem on 'mail sample 1'. The quick find bar was properly returning this message when searching for strings in its headers (like from, subject).
The body of the message was shown as garbage. It is marked as base64 but the contents do not look like base64, it is plain text. 

Can anybody knowledgeable with MIME check if that message is correctly encoded?

Comment 17

[Luis Menina]

(In reply to :aceman from comment #16)
> I do not see the problem on 'mail sample 1'. The quick find bar was properly
> returning this message when searching for strings in its headers (like from,
> subject).

As I said in previous comment, the message may be correct if the problem is in the indexation engine. When the bug happens, the message may just never show up in any search request.

> The body of the message was shown as garbage. It is marked as base64 but the
> contents do not look like base64, it is plain text.

Yes, it's plain text encoded in base64. The base64 part looks valid, as decoded by http://www.base64decode.org/

Comment 18

[:aceman]

Attachment: screenshot of 'mail sample 1' rendering

This is how I see the message being rendered. Is that correct display?

Comment 19

[Luis Menina]

Oh, it looks there's a problem with the eml files 1 & 2. As I couldn't receive these emails, my colleague forwarded them to me, as attached eml files. This seems to have altered the messages, as the content has been decoded to plain text, but is still marked as being base64 in the headers. Is that a bug ? He just selected the two emails, right clicked on them, and chose "forward as attachment" (or something similar) in the context menu.

Please give a look at email 3, that one is correct.

Comment 20

[:aceman]

Ok, now what exact string should I look for to see the bug (no search results)?

Comment 21

[Luis Menina]

Attachment: email sample 1

Comment 22

[Luis Menina]

Attachment: email sample 2

Comment 23

[Luis Menina]

(In reply to :aceman from comment #20)
> Ok, now what exact string should I look for to see the bug (no search
> results)?

Well anything you want. Lookup for "Project" for example. When the bug happens, the concerned emails just never show up, in any request to search in message body. Make sure you disable search on other criteria, like "subject", because these work correctly.

I tried several words contained in the email, but even with simple one-letter searches, these emails don't show up.

Comment 24

[:aceman]

OK, I can see the problem, the message is not matched.

Comment 25

[Luis Menina]

Any news on this?

Comment 26

[Filipus Klutiero]

Does this affect any mail which is not base64-encoded? If not, the summary should be changed to reflect that.
The summary should also feature the terms "false negatives" to ease finding this ticket.