697021 - "Body search of multipart mail" searches (i) data after close boundary of the mail and (ii) unix mbox separator/headers/payload of next mail. Body Search of multipart message should stop at "close boundary" and "end of mail".

Status: RESOLVED DUPLICATE of bug 1105937

(MailNews Core :: Search, defect)

Description

[WADA:World Anti-bad-Duping Agency]

Attachment: Mail folder file, with test mails

Spin off of bug 569009 comment #9.

Body Seach of multipart message doesn't stop at "close boundary" nor "end of mail". Searchs message headers and first body line of next mail.

Attached mail folder file cotains test mails;
(a) mail-000 : multipart/mixed mail, "?" doesn't exist in message source.
(b) mail-01/mail-02/mail-03, text/plain mail, 
    same message headers, single body line, different body line test.
  mail-01 : ?01 ?01 ?01
  mail-02 : ?02 ?02 ?02
  mail-03 : ?03 ?03 ?03
(c) mail-88/mail-99/mail-99X/mail-99Y, text/plain mail, 
    same message headers, two body line, similar body line text,
    message header is slightly different
  mail-88  : Same header as mail-01, two body lines.
             ?88 ?88 ?88
             ?04 ?04 ?04
  mail-99  : Same header as mail-01. ?04 is moved to first body line.
             ?99 ?99 ?04
             ?05 ?05 ?05
  mail-99X : UIDL: is added to mail-99.
             Same body lines as mail-99
             ?99 ?99 ?04
             ?05 ?05 ?05
  mail-99Y : Mime-Version: is removed from mail-99X.
            (UIDL: is added to mail-99, Mime-Version: is removed from Mail-99)
             Same body lines as mail-99
             ?99 ?99 ?04
             ?05 ?05 ?05
(d) mailend
    text/plain mail to see size of upper mail via "Order Received" column.

Comment 1

[WADA:World Anti-bad-Duping Agency]

[Steps to reproduce]

(0) Show "Order Received" column, sort in ascending order, for ease of testing.
    Disable auto-compact, to avoid interfere of testing by auto-compact.
    At search box, type ?, change "Filter message by:" to "Body only".
(1) Body filter searchs first body line of next mail.
(1-1) Search text : ?01 => mail-001 and mail-01 is shown
(1-2) Clear search text, Shift+Delete mail-01
      Search text : ?01 => mail-001 is still shown
      Clear search text, Compact folder, 
      Search text : ?02 => mail-001 and mail-02 is shown
(1-3) Clear search text, Shift+Delete mail-02
      Search text : ?02 => mail-001 is still shown
      Clear search text, Compact folder, 
      Search text : ?03 => mail-001 and mail-03 is shown
(1-4) Clear search text, Shift+Delete mail-03
      Search text : ?03 => mail-001 is still shown
      Clear search text, Compact folder, 
      Search text : ?88 => mail-001 and mail-88 is shown
      Search text : ?04 => mail-001 is not shown (second line is not searched)
(2) First line is surely searched, Second line is surely not searced.
(2-0) Clear search text, Shift+Delete mail-88, Compact folder
(2-1) Search text : ?99 => mail-001 and mail-99/mail-99X/mail-99Y
      Search text : ?04 => mail-001 and mail-99/mail-99X/mail-99Y
      Search text : ?05 => mail-001 is not shown, mail-99/mail-99X/mail-99Y only
(3) Message headers of next mail is searched.
    First line is searched or not depends on number of header lines of next mail.
(3-1) Search text : UIDL:         => no mail is shown
      Search text : Mime-Version: => mail-001 is shown
      Search text : ?04           => mail-001 and mail-99 is shown
(3-2) Clear search text, Shift+Delete mail-99, Compact folder
      Search text : UIDL:         => mail-001 is shown
      Search text : Mime-Version: => no mail is shown
      Search text : ?04           => mail-001 is not shown, mail-99X/Y only
                    (first line is not searched if one header is added. Why?) 
(3-3) Clear search text, Shift+Delete mail-99X, Compact folder
      Search text : UIDL:         => no mail is shown
      Search text : Mime-Version: => mail-001 is shown
      Search text : ?04           => mail-001 and mail-99Y is shown
                                     (first line is searched)
Note-1:
At any step, search for From:, To:, Subject:, X-Mozzila-Status: etc. returns mail-001.
Note-2:
"Gloda is enabled or disabled" looks irrelevanto to phenomena.

Because above is intentional test result with crafted mail, "search of first mail body line of next mail" may not occur with really sent mails.
However, if multipart mail, "search of message headers of next mail" occurs in any step of above test. I think it will occur on any multipart mail.
Body search and/or Gloda Indexer should terminate at "close boundary" of primary "Content-Type: multipart/xxx boundary=..." and/or "end of mail data". 

IIRC, there are some bugs for phenomenon of "body search searches message header". I guess such problem is above multipart mail case.

Comment 2

[WADA:World Anti-bad-Duping Agency]

Test result of Mail-Version: in step (3-2) should be; (sorry for spam) 
> (3-2) Clear search text, Shift+Delete mail-99, Compact folder
>       Search text : Mime-Version: => mail-001 is shown
>                                   (mail headers of next mail is surely searched)

Comment 3

[WADA:World Anti-bad-Duping Agency]

FYI. 
Following bugs are probably for phenomenon of "body search searches message header".
> bug 379988 : search body should not match words in MIME headers
> bug 576994 : "Body" quick filter option searches body and seemingly-random header values

Comment 4

[WADA:World Anti-bad-Duping Agency]

Same phenomena was observed by Virtual Folder(Saved search folder).
If Virtual Folder or Search, multiple conditions can be set. So next can be used to see whether multipart only problem or not;
  - Virtual folder A :     Body contains Subject:
                       and Content-Type contains multipart
  - Virtual folder B :     Body contains Subject:
                       and Content-Type doesn't contain multipart
And if Virtual Folder, multiple folders can be selected as search target folder. It'll make problem determination and testing for it easier.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Attachment: mail with message/rfc822 part(mail-001 is attached to mail)

mail-001 of above test is attached as message/rfc822 part to text mail, and saved by Send Later. 
Fortunatly, mail-001 had next wrong header by my mistake :-)
> Content-Disposiyion: attachment; filename=...
It is useful to see body search on message/rfc822 part, which is absolutely diffferent problem from this bug. 

Body search currently doesn't exclude data in message/rfc822 part from search target data. So, body search for "Content-Disposiyion: attachment; filename=" returns the mail to which mail-001 attached.
IIRC, this is known issue, but it might be bug report for message filter.

Comment 6

[WADA:World Anti-bad-Duping Agency]

FYI.
If mail is downloaded from POP3 server by Tb, X-UIDL: header is always written and its content is unique. So, this bug's phenomenon on message header of next mail is easily be observed.
(1) Show "Order Received" column of a folder, sort in ascending order.
(2) Choose a mail, view message source of a mail,
    and copy XUIDL: header line to clip board
(3) Search via context menu of the folder, Body contains, paste from clip board
    => search always returns upper mail of the chosen mail at step (2) 
Unique ID written in Received: header by server is also useful to see the phenomenon.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Attachment: Test case, with X-Test#: headers in next mail

Test case is modified to see number of searched lines in next mail.
(a) mail-000 : multipart/mixed mail, "?" doesn't exist in message source.
    Same as original test.
(b) mail-101, text/plain mail, 
    same as mail-9Y in original test, except next;
      X-Test#: ... headers are added after X-Mozilla-Keys: header,
      where # is "a" to "z" (26 headers are added to mail-9Y)

Test result.
  Search for next Unix Mbox separator line of mail-101 returned mail-000.
     From - Sat Jan 01 01:02:03 2011
  Search for headers before X-Testa: returned mail-000.
  Search for X-Testa: to X-Testh: returned mail-000.
  Search for X-Testi: to X-Testz: returned nothing.
  Serach for headers after X-Testz:, such as Message-ID:, returned nothing.

It seems that Tb searches predefined lines in next mail.
This is probably reason why problem report is usually for headers added by Tb and Received: header.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Body Search returns a mail if any line of "lines after close boundary of the mail" contains searching string.
Problem may be:
(i)   Body search searches data after close boundary of multipart mail.
(ii)  The "search lines after close boundary" doesn't stop search at end of mail,
      and searches several lines after end of mail.
(iii) Thus, top most several lines of next mail(unix mbox separator, headers,
      and payload, of next mail) is searched.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Test mails to see phenomenon of "Body Search searches lines after close boundary".
  attachment 573061 [details] (attached to bug 481616 comment #12)
> Case-1A: text/plain, Case-2A: multipart/mixed, Case-3A: multiprt/alternative under multipart/mixed
I intentionally added "lines after close boundary" to see "search unix mbox separator/header/body of next mail" is what kind of problem.

(i) Body Search serches top part of next mail.
  Body search for From - Wed Nov 02 19:02:02 2011 => Test mail of Case-2A
  (Unix mbox separator for Case-3A is searched)
  Body search for Case-3A => Test mail of Case-2A
  (Subject: header of Case-3A is searched)
(ii) Body Search serches lines after close boundary.
Case-2A or Case-3A is returned to Body Search for next sring which is placed after close boundary.
> Case2AAfterCloseBoundaryKeyword=3DValue
> Case3AAfterCloseBoundaryOfAltKeyword=3DValue
> Case3AAfterCloseBoundaryKeyword=3DValue
(iii) "Close boundary of nested multipart"(not top level close boundary) is searched by Body Search.
Body Search for AltBdy, --AltBdy-- returns mail of Case-3A.
No mail is found if search for MixBdy which is top level boundary.

Phenomenon of "search header/body of next mail" may be one like next;
  "incorrect search of lines after close boundary" uses "incorrect number of
  lines in a part", then, over end of the mail, it searches several lines of
  next mail which starts with unix mbox separator and message header.

If such problem, why -AltBdy--(close bounary of nested multipart) is searched? Quote-printable relevant phenomenon?

Comment 10

[WADA:World Anti-bad-Duping Agency]

FYI.
For "=3D" case. If text contains string of "=" + "two hexadecimal digits", when non-quoted-printable, bug 667854 occurs, and when quoted-printable(=3D is encoded "="), bug 481616 occurs.

Comment 12

[Honza Bambas (:mayhemer)]

-> me

Comment 13

[Thomas D. (:thomas8)]

Honza (assignee), any news from your looking into this bug?

Comment 14

[Thomas D. (:thomas8)]

I find it very odd that WADA is serving reduced easy testcases for bad and confusing bugs like this one on a silver tablet and after more than half a year, we don't even have a single developer comment.

Comment 15

[:aceman]

Sometimes WADA's genius is too much for us.

But yes, it would be nice if somebody looked at it.

Comment 16

[Thomas D. (:thomas8)]

Attachment: testcase X2 (mail folder file with test messages)

This is same as testcase X1 mailfolder from attachment 569296 [details], but I corrected a spelling mistake (Content-Disposi*y*ion) in Content-Disposition header of "mail-000 multipart-mixed" which causes attachments to not show in attachment pane. Inconsequential for this bug.

Comment 17

[Thomas D. (:thomas8)]

(In reply to :aceman from comment #15)
> Sometimes WADA's genius is too much for us.

The trick is *not* to try to understand all the details and not to worry about the sometimes difficult wording too much. Just try the testcases and follow one or two instructions, and the bug is very evident... ;)

Comment 18

[Honza Bambas (:mayhemer)]

I don't intend to work on this bug any more, my priorities has unfortunately changed.

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #15)
> it would be nice if somebody looked at it.

Following is copy of bug 576994 comment #15.
This is phenomenon written in bug summary, and phenomenon what I tried to see by my test mails and my test procedures.

Phenomenon of bug 697021.
> <----- mail-1 (multipart/xxx) -----><------------- mail-2 ------------->
> <-- head --><-------- body --------><-------- head --------><-- body -->
>             <------ Body search by bug 697021 ------>
If mail-2 is crafted mail, following is observed.
> <----- mail-1 (multipart/xxx) -----><------------- mail-2 ------------->
> <-- head --><-------- body --------><- head -><-------- body ---------->
>             <------ Body search by bug 697021 ------>

Comment 20

[WADA:World Anti-bad-Duping Agency]

It looks that "understanding of this bug" is impossible due to "difficult wording too much" by me and broken English by me.
As no one can understand this bug, "keeping this bugopen" is absolutely uselesss.
Closing this bug as dup of Bug 1105937, which is reported for phenomeno in actual environment by people of soundplant.org using normal English.

Comment 21

[WADA:World Anti-bad-Duping Agency]

Attachment: Test case, with many X-Test?#: in subpart of multipart, with many X-Test#: headers in next mail

Body Search.
> http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgLocalSearch.cpp#491
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsMsgHdr.cpp#539
> http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgSearchTerm.cpp#913
>   nsMsgSearchTerm::MatchBody (nsIMsgSearchScopeTerm *scope, uint64_t offset, uint32_t length /*in lines*/, const char *folderCharset,
>                                                       nsIMsgDBHdr *msg, nsIMsgDatabase* db, bool *pResult)

- Body Search is executed on "number of line of message payload" basis.
- HTML tag is skipped.
- message header in sub part of multipart is skipped.
So, many X-Test?#: header is also added to sub part of multipart in addition to "X-Test#: header in sencond mail".

As I expected, (i) body search for "any header line of 2nd / 3rd mail" returns 1st mail., and (ii)  serch for "body line of 2nd mail" returns 1st mail & 2nd mail and body search for "body line of 3rd mail" returns 1st mail & 3rd mail.

Searched lines == perhaps appoximately "number of mail payload lines of multipart mail" 
                                                                + "number of message header lines in sub part of the multipart mail"
When "message header line in sub part of multipart mail" is skipped, processed line count is not incremented?

Comment 22

[WADA:World Anti-bad-Duping Agency]

Attachment: Test mails : any string in mai#1/mail#2 returns mail#0 ( mail#0 = many X-Test?#: in subpart of multipart, mail#1 = many X-Test#, mail#2 = small text/plain )

Updated version of test mail attched to previous comment.

Any string in line after close boundary of mail#0 returns mail#0.
This is same when mail#1/mail#2 is deleted(Note: before Compact).