481616 - Local searching message fails when "=" is in the body, because quoted-printable text is searched as plain text even though "=" is encoded as "=3D" if quoted-printable (local search only, not IMAP online search)

Status: RESOLVED WORKSFORME

(MailNews Core :: Search, defect)

Description

[Stefan]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6
Build Identifier: 20081209

The "Entire Message" quick search as well as the "Body" search function fails to find any text in the body entered directly after a "=" unless the "=" is included in the search! (!)

Reproducible: Always

Steps to Reproduce:
1. Send an e-mail to yourself with the text "test=123".
2. Recieve the mail and try to find it by using the search string "123" using either "Search Messages" or the "Entire Message" quick search. 
3. You will notice that the search functions fails to find the message! However it works if you use the search string "=123"!
Actual Results:  
Actual Results: BOTY (Bug of the Year).

Expected Results:  
"What happened after you performed the steps above?", I, uh, screamed?

"What should the software have done instead?", well how about finding my freaking e-mail? Thanks.

Comment 1

[Ludovic Hirlimann [:Usul]]

this works for me with Gecko/20090304 Shredder/3.0b3pre.

>Build Identifier: 20081209
Stefan you seem to be running either a 2.x series or an old version of trunk (if that's the case please update to http://stage.mozillamessaging.com/en-US/thunderbird/early_releases/downloads/ where it works.)

Can you confirm the version of thunderbird used ?

Comment 2

[Stefan]

Thanks for your quick reply Ludovic! :) I was just too busy at work to reply until now...
Yes I am indeed running Thunderbird 2.0.0.19 (20081209). It is great that you've found that it works in the TB 3 beta, I guess I should give that a try (but not on my work computer). I'll let you know if the problem remains.

Comment 3

[Stefan]

Well now I have tried both the regular Thunderbird 3.0b2 version and the latest of that Shredder thing (Gecko/20090305 3.0b3pre) from here:
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/

Result: Same as Thunderbird 2.0, it does not work!
I tried this on my Macbook now instead of my Windows work computer.

One strange thing I noticed is that Thunderbird finds the message with the "test=123" body text if I search for "3" but "1" does not work, "2" does not work and "12" does not work.

Somebody should hunt down and kill this bug before TB3 is released!

Comment 4

[Phil Ringnalda (:philor)]

Can you attach an email (from Save As - File) where you see this? Does it actually happen for all emails, including tests you send yourself from Thunderbird? POP, or IMAP? Are the messages plain text, HTML, or both?

If you aren't finding "=12" but are finding "3", then that makes it sound like search thinks that the "=12" is actually a quoted-printable encoded DC2 control character, but I'm with Ludovic, I can't find any combination of mail contents and formats and server types which will confuse search that way.

Comment 5

[Stefan]

Thanks for your reply Phil. I'd prefer not to attach an e-mail if possible...
I have now found that the search works fine with e-mails that I send to my POP3 mail from Yahoo Mail, but nothing that I send myself from Thunderbird works! I have tried sending from different POP3 addresses and from different SMTP servers.

The mails from Yahoo have:
Content-Type: multipart/alternative; boundary="0-466073481-1236334059=:783"

The mails from Thunderbird have either of:
Content-Type: text/plain; charset="iso-8859-1"
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Type: text/html; charset=ISO-8859-1

Today I have been testing with TB 2 for Windows XP. TB 3 on my Mac which also didn't work yesterday was installed for the first time by the way with no other version ever installed so the settings were all default.

I think you are on to something in that it may be interpreted as a control character, the question is why.

Comment 6

[Ludovic Hirlimann [:Usul]]

Attachment: Test case

I can reproduce when sending the email from my yahoo account.

Comment 7

[Stefan]

Awesome, thanks Ludovic! I'm not sure how to import your mail into Thunderbird but great that you were able to reproduce the problem.
I have tried sending myself both plain text and RTF messages from Yahoo and both work for me so it's strange!

Comment 8

[Wayne Mery (:wsmwk)]

xref  Bug 293823 -  "Search messages" doesn't work with unicode (even though "find in this message..." does) 

I'd be shocked if this isn't a duplicate, but I'm not sure if that's the best one to dupe to. Perhaps some leads in http://tinyurl.com/c9xqkv (seems like that list is too small. is there a better search?)

Comment 9

[Wayne Mery (:wsmwk)]

all the lovely issues mentioning yahoo http://tinyurl.com/cnrwu6

Comment 10

[Stefan]

Wayne, one of the e-mails I sent myself from Yahoo which works with the search is encoded in quoted-printable so I'm going to restore my brutally hijacked Summary now thank you very much.

I don't find anything relevant in the bugs Wayne linked to but I have noticed now that all of the mails that does not work seems to have 7bit Content-Transfer-Encoding. Searching these messages for the string "=3D" which should be the encoded version of the equal sign does not give a result but "=3D" does give a result on the Yahoo mails which works. I'm not sure if that means anything.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to Ludovic Hirlimann [:Usul] from comment #6)
> Created attachment 365859 [details]
> Test case

Not multipart mail(text/plain mail), and message body is quoted-printable encoded.
> Content-Type: text/plain; charset=utf-8
> Content-Transfer-Encoding: quoted-printable

This bug looks quoted-printable version of bug 132340 on base64 encoded & non-multipart mail(text/xxx mail), because string of "=3D" is searched by Tb instead of decoded "=".
Do you still see this bug in recent Tb 7?

See bug 37031 (and many dups of the bug) for "base64 encoded attachment part of multipart mail" case, please. Similar problem to this bug & bug 37031 may occur on "quoted-printable encoded attachment part of multipart mail".
Please carefully separate (a) non-multipart mail case, (b) message body part(never attachment) of multipart mail case, and (c) sub parts(attachment part usually) of multipart mail case.
Atached mail is case of (a).

Comment 12

[WADA:World Anti-bad-Duping Agency]

Attachment: Test mails(text/plain case, multipart/mixed case, multiprt/alternative under multipart/mixed case)

Problem occurs on any of (a) non-multipart mail, (b) message body part(never attachment) of multipart mail, and (c) sub parts(attachment part usually) of multipart mail. Any valid body text which involves = can be found by Body Search for "...=3D..." only.
It looks for me next.
                                  base64                quoted-printable
(a) body of non-multipart  mail   fixed by bug 132340   found by this bug
(b) body part of multipart mail   fixed by bug 132340   found by this bug
(c) sub parts of multipart mail   bug 37031             found by this bug

In additio to above, next is also observed, because multipart/mixed mail is contained and because I intentionally added "lines after close boundary" for testing of other cases and other bugs.
(i) Phenomenon found by Bug 697021. Body Search serches top part of next mail.
  Body search for Case-3A => Test mail of Case-2A is returned
  (Subject: line of Case-3A is searched)
(ii) Body Search serches lines after close boundary.
Case-2A or Case-3A is returned to Body Search for next sring placed after close boundary.
> Case2AAfterCloseBoundaryKeyword=3DValue
> Case3AAfterCloseBoundaryOfAltKeyword=3DValue
> Case3AAfterCloseBoundaryKeyword=3DValue
(iii) "Close boundary of nested multipart" is searched by Body Search.
Body Search for AltBdy, --AltBdy-- returns mail of Case-3A.
No mail is found if search for MixBdy which is top level boundary.

(ii) & (iii) are perhaps one of causes of Bug 697021 or a variant of phenomenon by Bug 697021.

Comment 13

[WADA:World Anti-bad-Duping Agency]

I couldn't see this bug's problem with Gmail IMAP and offline-use=off folder(online search is forced. Body doesn't appear as criteria if Search of context menu or Edit/Find/Search Messages, unless "online search" is requested). Mail is returned as expected by search like "Body contains Keyword=Value".

If offline-use=On(Gloda=on/off is relevant?), search is executed locally by Tb, then this bug occurs.

However, all mails were returned by Body Search for --AltBdy-- and --MixBdy-- (online search of body by Gmail IMAP). This may be Gmail IMAP bug, but problem of Tb like "Tb shows all mails when somthing wrong occurred internally" because following excepcion was shown at Error Console during test of Body Search.
> Error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.getStringProperty]"
>  nsresult: "0x80004005 (NS_ERROR_FAILURE)"
>  location: "JS frame :: chrome://messenger/content/folderPane.js :: getSmartFolderName :: line 2467"  data: no]
> Source File: chrome://messenger/content/folderPane.js
Line: 2469
This is different problem from this bug. I'll check it.

Comment 14

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 15

[Jorg K (CEST = GMT+2)]

Many broken things were fixed in body search. This will work now.