404255 - Consider using UTF-8 when searching inside message body (IMAP online search) to avoid search failure

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Nikolay Shopik]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9
Build Identifier: version 2.0.0.9 (20071031)

I'm using English version of TB. If I'm trying to search inside entire message using Cyrillic symbols it doesn't return any messages, because it incorrectly select default search encoding. Here log captured from Wireshark. If you change this parameter to something other than ISO-8859-1 it start working Tools->Options->Display->Formating->Fonts->Incoming Mail->

5 uid SEARCH CHARSET ISO-8859-1 UNDELETED BODY "B5AB"
* SEARCH
5 OK Search completed. 

Reproducible: Always

Steps to Reproduce:
1.Create message with with non ASCI symbols
2.Tools->Options->Display->Formating->Fonts->Incoming Mail->ISO-8859-1 must be selected (default in english version of TB)
3.Make search
4.nothing returns
Actual Results:  
It probably should aware of input charters or use UTF-7 encoding in search always not depends on this setting.

Comment 1

[Nikolay Shopik]

Reproducible on trunk 3.0a1pre (2008040204).
Windows XP SP2
Consider using UTF-8 or UTF-7 by default in english version of TB too.

Comment 2

[Magnus Melin [:mkmelin]]

Which kind of search, from the search dialog or using quick search? IMAP/POP?

Comment 3

[Nikolay Shopik]

quick search, IMAP, if you have this folder where you searching is available for offline there no such problem because no request is sent to server it searching locally.

Comment 4

[WADA:World Anti-bad-Duping Agency]

WORKSFORME, Tb trunk(2008041103 build on Japanese MS Win-XP) with mail of Content-Type:text/plain;charset=iso-2022-jp on Gmail IMAP server.
"CHARSET ISO-2022-JP" was specified on search command.

Character encoding is specified in mail header explicitly?
Search of Content-Type:text/html; part with charset for Cyrillic in <meta> or lang attribute, in a mutipart/alternative mail?

Comment 5

[WADA:World Anti-bad-Duping Agency]

Oh, sorry. I seems to have misunderstood problem, and, iso-2022-jp was set in folder property of Gmail IMAP Inbox.
 (a) Tb trunk(en-US) on Japanese MS Win 
 (b) folder/properties/General/Default Character Encoding == iso-8859-1
 (c) Search in context menu of IMAP folder, "BODY contains"
 (d) "日本語" (Japanese kanji for "Japanese") in search word
Following was sent to Gmail IMAP server.
> uid SEARCH CHARSET ISO-8859-1 UNDELETED BODY {3}

Confirmed.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Note:
After the command, data of 0xE52C9E was sent by Tb for +go ahead in my environment.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Workaround:
If fall back charset in mail display is not needed(not_so_many mails with no charset/mail data in non-asci), set UTF-8 in Folder Property/Default Character Encoding, or charset which is usually used for text editing under your OS.
If fall back charset in mail display is required(many mails with no charset/mail data in non-asci, or many mails with incorrect charset), and narrower charset such as iso-8859-1 is required for fall back charset, split of mail folder is a possible circumvention, but I can say nothing for this case.

Adding "IMAP SEARCH CHARSET" in bug summary for ease of search.
Nikolay Shopik(bug opener), please change lengthy summary by me to appropriate one.

Comment 8

[Nikolay Shopik]

you mean to this one - CHARSET of IMAP SEARCH is set to 'Default Character Encoding' of folder property.
Right?

Comment 9

[WADA:World Anti-bad-Duping Agency]

> Right?
No. It's simply a short explanation of my test result. I only hope combination of IMAP,SEARCH,CHARSET that is suggestive IMAP command of "uid SEARCH CHARSET", for ease of search & understanding problem. I think "incorrectly select encoding" or your request of "Consider using UTF-8" is also important words.

Comment 10

[Wayne Mery (:wsmwk)]

Nikolay, is this really "minor" to you?  wanted-TB3?

Comment 11

[Nikolay Shopik]

I don't mind to escalate it to normal, minor just because easy workaround. UTF8 aware app is very important for localized version.

Comment 12

[Wayne Mery (:wsmwk)]

Martin, Any calendar implications? 

(In reply to comment #11)
> UTF8 aware app is very important for localized version.

=> major would seem appropriate then

Comment 13

[Martin Schröder [:mschroeder]]

(In reply to comment #12)
> Martin, Any calendar implications?

Wayne, none that I know of.

Comment 14

[Wayne Mery (:wsmwk)]

would this be delight for intl users?

Comment 15

[David :Bienvenu]

we'd love to see a patch for this, but it's not blocking 3.0

Comment 18

[Wayne Mery (:wsmwk)]

Jorg, Does this ring a bell - sounds similar to bug that may have had activity in past year

Comment 19

[Jorg K (CEST = GMT+2)]

Quick search/regular search in bodies in - EDIT: local/offline folders - was fixed in bug 1427124.

EDIT: Let's pronounce this fixed by bug 1427124. Oops, the bug is about "online search".

Comment 20

[Magnus Melin [:mkmelin]]

That was only local search, this is the online case.

Comment 21

[Jorg K (CEST = GMT+2)]

Right, I missed "online". Does that "search on server" work at all for bodies? Oops, I don't actually close the bug.

Comment 22

[Magnus Melin [:mkmelin]]

Of course (depending on the server). Many servers have the texts indexed so then body search is very fast server side.

Comment 24

[Alfred Peters [:infofrommozilla]]

(In reply to Nikolay Shopik from comment #8)

you mean to this one - CHARSET of IMAP SEARCH is set to 'Default Character
Encoding' of folder property.
Right?

Exactly. See (bug 1669775 comment #12)

Comment 25

[Ian Eales]

78.4.0
US English
IMAP and Local Folders
Searching 'Body' in Search Folders... returns nothing.