1434020 - Quick Filter Toolbar and Message Search don't find content in bodies of attached messages

Status: RESOLVED FIXED

(MailNews Core :: Search, defect)

Description

[Gil Neiger]

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36

Steps to reproduce:

After installing 52.6.0, I tried using the Quick Filter Toolbar and doing searches for messages.


Actual results:

Some but not all matching messages were listed.


Expected results:

All matching messages should be listed.

I tried deleting all my msf files and rebuilding global-messages-db.sqlite.  It did not help.

I then rolled back to 52.5.2, and everything was fine.

Reinstalling 52.6.0 again caused some messages to be missed; rolling back again to 52.5.2 again fixed the problem.

The release notes for 52.6.0 includes the following items:  "Searching message bodies of messages in local folders, including filter and quick filter operations, not working reliably: Content not found in base64-encode message parts, non-ASCII text not found and false positives found."

It seems that this change may have removed false positives but it introduced false negatives.

Comment 1

[Jorg K (CEST = GMT+2)]

(In reply to Gil Neiger from comment #0)
> It seems that this change may have removed false positives but it introduced
> false negatives.
That would be unfortunate but can't be ruled out. Can you attach a message here ("Attach File" above) and tell us what content wasn't found? Please remove any personal information before attaching the message.

Comment 2

[Gil Neiger]

Attachment: Email that Quick Filter finds differently on different versions

Comment 3

[Gil Neiger]

I attach here an email that is found disparately on a Quick Filter of the string "Cornell" in the Body.

52.5.2 finds it just fine.  52.6.0 does not find it.

Comment 4

[Jorg K (CEST = GMT+2)]

Thanks for reporting and thanks for providing the sample.

Strictly speaking the body of the e-mail doesn't contain the word "cornell" since the body is simply this:

===
The enclosed message, found in the PODC  mailbox and shown under the spool ID
380297011 in the system log, has been identified as a possible delivery error
notice for  the following  reason: "X-Report-Type:" field  found in  the mail
header.
===

"cornell" is found in the body of an attached message. Looking at the code I can see that the original intent was to search through bodies of attached message as well, so something indeed stopped working from 52.5.2 to 52.6.

I'll see whether I can get this fixed for TB 52.7.

Comment 5

[Gil Neiger]

Sorry to have missed this detail - I view attachments inline, so I hadn't focused on the fact that this was limited to attachments.

Thanks for addressing this.  I manage a mailing list and get these "bounce" messages (with attached mails) all the time, and it's very helpful for me to be able to search for text inside of the attached mails.

Comment 6

[Jorg K (CEST = GMT+2)]

Attachment: 1434020-body-search-attached.patch - (v1) WIP

Here I'm adding three test cases of a message with an attached message of various MIME structures, including one with base64 encoded parts. The automatic test shows that nothing is found in the body of the attached messages.

So the next step will be to fix the code so the content is actually found and reflect that in the expected test result (of course).

Comment 7

[Jorg K (CEST = GMT+2)]

Attachment: 1434020-body-search-attached.patch - (v2) WIP

Added more test cases. Theoretically we'd have to all of the 30 message we test as attachments. But let's see whether we can get by with fewer tests.

Comment 8

[Jorg K (CEST = GMT+2)]

Attachment: 1434020-body-search-attached.patch (v3)

OK, here is the fix. It took a quiet Sunday evening to look into it.

Comment 9

[Jorg K (CEST = GMT+2)]

Attachment: 1434020-body-search-attached.patch (v3b)

Minor tweak to re-initialisation when encountering message/rfc822. We assume the part is plaintext unless proven otherwise. That caters for plaintext messages lacking the Content-Type: header. This also aligns this initialisation with the one in the constructor.

Comment 10

[:aceman]

Comment on attachment 8948329 [details] [diff] [review]
1434020-body-search-attached.patch (v3b)

Review of attachment 8948329 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, works for me. And thanks for new test cases!

Comment 11

[Jorg K (CEST = GMT+2)]

(In reply to :aceman from comment #10)
> And thanks for new test cases!
Those are very important. Had we had those before, this regression wouldn't have happened. Also they document nicely what works (and what isn't covered).

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c6a63bb68388
make body search also look in bodies of attached messages. r=aceman

Comment 13

[Jorg K (CEST = GMT+2)]

Beta (TB 59):
https://hg.mozilla.org/releases/comm-beta/rev/8608e5bbbaf69df69869668b8af876d645a2ac98

Comment 14

[Jorg K (CEST = GMT+2)]

TB 52.7 ESR:
https://hg.mozilla.org/releases/comm-esr52/rev/c1d839ba63521bf5a5d64ec65b128cf18e4d4efb

Comment 15

[jmdyck]

I am experiencing similar behavior, but the text in question is not in an attachment per se, but in multipart/alternative sections. Would this bug also cause that text to not be found by Search?

Comment 16

[Jorg K (CEST = GMT+2)]

This bug was fixed a while ago. We have tests in our system that check multipart messages. If this doesn't work for you, please file a new bug and attach a sample message that isn't found.

Comment 17

[jmdyck]

> This bug was fixed a while ago.1478828

Yes, but I'm running 52.9.1 (Ubuntu 18.04's newest version), so "RESOLVED FIXED in Thunderbird 60.0" suggests that I don't have that bugfix yet. I was trying to ascertain whether or not my observed behavior was due to my lack-of-bugfix. Anyhow, I've filed bug 1478828. Thanks!