219190 - Search messages always returns no results (news)

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: NNTP, defect)

Description

[Mat Dodgson]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.5a) Gecko/20030728 Mozilla Firebird/0.6.1
Build Identifier: 

Search messages always returns no results (news).

Reproducible: Always

Steps to Reproduce:
1.Subscribe to discussion.epoc.c++ on publicnews.symbiandevnet.com
2.Downloaded all headers (19000)
3.Tools -> search messages
4.Put anything in the search box
5.Always no results in the bottom
Actual Results:  
No results from search

Expected Results:  
Should have many results. Search for the letter 'a' should have perhaps 10,000
results.

Comment 1

[Mat Dodgson]

Perhaps the problem only occurs when searching a large list of messages? Anyway
the problem can be reproduced thus:

1. subscribe to "news://publicnews.symbiandevnet.com/discussion.epoc.C++"
2. download all headers (22000 or there abouts)
3. search on subject for anything. No results.

Problem exists in firebird 0.3 and 0.4.

Comment 2

[Allan Bressler]

I don't beleive it's the fact that there are a large amount of messages.  I have
a news account (news.superwaba.net) that has one group with many messages
(approx 20,000) and one with 500 and I can't get results returned from either one.

However, I do have another account (news.eclipse.org) that has a few groups with
around 10,000 messages and I can search those fine.

I'm using Thunderbird 0.5.

Comment 3

[MoinMan]

Confirm on Mozilla Thunderbird 0.5+ (20040318) and varies newsgroup accounts.

Comment 4

[Todd Ross]

Alright, here's what I've dug up.

Using netcat, I built a tunnel between Thunderbird and
publicnews.symbiandevnet.com and attempted a search for 'a'.  This returned:

[todd@color:/home/todd]> cat publicnews.symbiandevnet.com.log
< 00000000 32 30 30 20 4e 4e 54 50 20 53 65 72 76 69 63 65 # 200 NNTP Service
< 00000010 20 35 2e 30 30 2e 30 39 38 34 20 56 65 72 73 69 #  5.00.0984 Versi
< 00000020 6f 6e 3a 20 35 2e 30 2e 32 31 39 35 2e 36 37 30 # on: 5.0.2195.670
< 00000030 32 20 50 6f 73 74 69 6e 67 20 41 6c 6c 6f 77 65 # 2 Posting Allowe
< 00000040 64 20 0d 0a                                     # d ..
> 00000000 4d 4f 44 45 20 52 45 41 44 45 52 0d 0a          # MODE READER..
< 00000044 32 30 30 20 4e 4e 54 50 20 53 65 72 76 69 63 65 # 200 NNTP Service
< 00000054 20 35 2e 30 30 2e 30 39 38 34 20 56 65 72 73 69 #  5.00.0984 Versi
< 00000064 6f 6e 3a 20 35 2e 30 2e 32 31 39 35 2e 36 37 30 # on: 5.0.2195.670
< 00000074 32 20 50 6f 73 74 69 6e 67 20 41 6c 6c 6f 77 65 # 2 Posting Allowe
< 00000084 64 20 0d 0a                                     # d ..
> 0000000d 47 52 4f 55 50 20 64 69 73 63 75 73 73 69 6f 6e # GROUP discussion
> 0000001d 2e 65 70 6f 63 2e 6a 61 76 61 0d 0a             # .epoc.java..
< 00000088 32 31 31 20 34 34 34 33 20 32 39 37 20 38 37 36 # 211 4443 297 876
< 00000098 30 20 64 69 73 63 75 73 73 69 6f 6e 2e 65 70 6f # 0 discussion.epo
< 000000a8 63 2e 6a 61 76 61 0d 0a                         # c.java..
> 00000029 58 50 41 54 20 53 55 42 4a 45 43 54 20 20 31 2d # XPAT SUBJECT  1-
> 00000039 20 2a 5b 41 61 5d 2a 0d 0a                      #  *[Aa]*..
< 000000b0 32 32 31 20 48 65 61 64 65 72 73 20 66 6f 6c 6c # 221 Headers foll
< 000000c0 6f 77 0d 0a 2e 0d 0a                            # ow.....

-----

For comparison sake, let's try another public news server,
freenews.netfront.net, which returns:

[todd@color:/home/todd]> cat freenews.netfront.net.log
< 00000000 32 30 30 20 6e 65 77 73 2e 6e 65 74 66 72 6f 6e # 200 news.netfron
< 00000010 74 2e 6e 65 74 20 49 6e 74 65 72 4e 65 74 4e 65 # t.net InterNetNe
< 00000020 77 73 20 4e 4e 52 50 20 73 65 72 76 65 72 20 49 # ws NNRP server I
< 00000030 4e 4e 20 32 2e 33 2e 35 20 72 65 61 64 79 20 28 # NN 2.3.5 ready (
< 00000040 70 6f 73 74 69 6e 67 20 6f 6b 29 2e 0d 0a       # posting ok)...
> 00000000 4d 4f 44 45 20 52 45 41 44 45 52 0d 0a          # MODE READER..
< 0000004e 32 30 30 20 6e 65 77 73 2e 6e 65 74 66 72 6f 6e # 200 news.netfron
< 0000005e 74 2e 6e 65 74 20 49 6e 74 65 72 4e 65 74 4e 65 # t.net InterNetNe
< 0000006e 77 73 20 4e 4e 52 50 20 73 65 72 76 65 72 20 49 # ws NNRP server I
< 0000007e 4e 4e 20 32 2e 33 2e 35 20 72 65 61 64 79 20 28 # NN 2.3.5 ready (
< 0000008e 70 6f 73 74 69 6e 67 20 6f 6b 29 2e 0d 0a       # posting ok)...
> 0000000d 47 52 4f 55 50 20 33 64 66 78 2e 64 65 76 65 6c # GROUP 3dfx.devel
> 0000001d 6f 70 65 72 2e 6c 69 6e 75 78 0d 0a             # oper.linux..
< 0000009c 32 31 31 20 31 32 20 31 37 30 20 31 38 31 20 33 # 211 12 170 181 3
< 000000ac 64 66 78 2e 64 65 76 65 6c 6f 70 65 72 2e 6c 69 # dfx.developer.li
< 000000bc 6e 75 78 0d 0a                                  # nux..
> 00000029 58 50 41 54 20 53 55 42 4a 45 43 54 20 20 31 2d # XPAT SUBJECT  1-
> 00000039 20 2a 5b 41 61 5d 2a 0d 0a                      #  *[Aa]*..
< 000000c1 32 32 31 20 53 55 42 4a 45 43 54 20 6d 61 74 63 # 221 SUBJECT matc
< 000000d1 68 65 73 20 66 6f 6c 6c 6f 77 2e 0d 0a          # hes follow...
< 000000de 31 37 35 20 45 61 72 6e 20 63 61 73 68 20 66 72 # 175 Earn cash fr
< 000000ee 6f 6d 20 69 6e 74 65 72 6e 65 74 0d 0a 31 37 36 # om internet..176
< 000000fe 20 53 41 56 45 20 4f 55 52 20 53 43 48 4f 4f 4c #  SAVE OUR SCHOOL
< 0000010e 2c 20 50 4c 45 41 53 45 0d 0a 31 37 37 20 49 74 # , PLEASE..177 It
< 0000011e 27 73 20 48 61 6d 20 6e 6f 74 20 53 70 61 6d 21 # 's Ham not Spam!
< 0000012e 0d 0a 31 37 38 20 49 74 27 73 20 48 61 6d 20 6e # ..178 It's Ham n
< 0000013e 6f 74 20 53 70 61 6d 21 0d 0a 31 37 39 20 49 74 # ot Spam!..179 It
< 0000014e 27 73 20 48 61 6d 20 6e 6f 74 20 53 70 61 6d 21 # 's Ham not Spam!
< 0000015e 0d 0a 31 38 30 20 53 41 56 45 20 4f 55 52 20 53 # ..180 SAVE OUR S
< 0000016e 43 48 4f 4f 4c 2c 20 50 4c 45 41 53 45 0d 0a 31 # CHOOL, PLEASE..1
< 0000017e 38 31 20 4d 41 4b 45 20 24 24 24 20 42 59 20 57 # 81 MAKE $$$ BY W
< 0000018e 4f 52 4b 49 4e 47 20 46 52 4f 4d 20 48 4f 4d 45 # ORKING FROM HOME
< 0000019e 21 21 21 21 0d 0a 2e 0d 0a                      # !!!!.....

-----

What have we learned?  Executing "XPAT SUBJECT  1- *[Aa]*" on
publicnews.symbiandevnet.com returns nothing while freenews.netfront.net returns
its results.  It seems to me that this problem is publicnews.symbiandevnet.com
specific.

I don't know that there's anything to "fix" in regards to Thunderbirds behavior.
 I'll leave that answer to someone who's more familiar with NNTP.

Comment 5

[Koike Kazuhiko]

*** Bug 243120 has been marked as a duplicate of this bug. ***

Comment 6

[Dave Hooton]

(In reply to comment #4)
> I don't know that there's anything to "fix" in regards to Thunderbirds behavior.
>  I'll leave that answer to someone who's more familiar with NNTP.

Thunderbird 0.8 returns TOO MANY results in news.logos.com. Try searching
"SENDER> CONTAINS> hooton" in General NG. Other Sender names are returned
(Win98SE & Win XP).

Comment 7

[J.Wilkes]

I experience a similar problem with ThunderBird 0.9, which manifest itself like
this:
I can search and find results, but a "Subject contains..." works only if I treat
it like it were "Subject starts with". (server with closed user group)

From the previous replies I assume this problem lies on the news server end
rather than in Thunderbird. If Thunderbird implements its search by just
interfacing with the search function of the server, these problems are indeed
outside Thunderbird.
However, it is difficult for most users to fix a percieved problem by patching
the server side.

Is there no local header store/cache which could/should be searched in?

Comment 8

[Jerry Carlin]

I get "A News (NNTP) error occured: xpat not supported.

Comment 9

[Dave Hooton]

(In reply to comment #6)
> Thunderbird 0.8 returns TOO MANY results in news.logos.com. Try searching
> "SENDER> CONTAINS> hooton" in General NG. Other Sender names are returned
> (Win98SE & Win XP).

Now I get too few hits in TB 1.0. It returns ZERO or a few hits that DO NOT
match my criteria in "Sender > begins with" and "Subject > begins with"!!!

Comment 10

[Erik Wikström]

For the same server and the same groups I can in Outlook Express seach for
messages and get results whereas I in Thunderbird 1.0.2 (20050317) can't.
Regerdeless of server-support I should at least allways be able to search for
messages containing/having a specific subject and get a result if I have the
message-header stored locally. If I also have the whole body of the message then
I should be able to used all search-options. 

It works for mail and I can see no reason why it shouldn't work for news, should
the server support searching that's nice but it shouldn't be necessary.

Comment 11

[Erik Wikström]

I've found kind of a work-around for this problem but it's not perfect as it only allows you to search messages whos header you've downloaded. The trick is to go offline and then search, the search will then be done on the downloaded headers.

Comment 12

[J.B. Nicholson]

What do you value more: giving the user accurate and complete search returns or doing what's more technically expedient?  I think that question ought to drive how to handle this bug.

It's not wise to exclusively rely on XPAT support because when XPAT isn't supported on an NNTP server, the user is left scratching their head wondering why the search is broken.  Novices won't do protocol traces, so novices won't figure out that XPAT support is required server-side, and then be satisfied to restrict themselves to doing subject and sender searches only on NNTP servers that support XPAT.

Also, Thunderbird's use of using:

XPAT SUBJECT 1- <subject query>

only works correctly if the user is searching only for subjects only.  The default search in TB is "Subject or Sender" which requires *two* XPAT commands (at a minimum), one on the Subject header, and one on the From header, then merging the results from both XPAT commands.

Searching the entire message can't be done with XPAT (since XPAT only searches headers, and perhaps only certain headers at that on some servers).  Searching the entire message body will be expensive with an NNTP host because it requires downloading the entire article -- headers & body -- for each article in the current newsgroup.

So, I don't think that the TB search facility was really built to work well with NNTP servers.  But if that search facility is what TB is going to offer with NNTP servers too, it might as well work correctly and have a good fallback mechanism for when XPAT is unavailable (perhaps also when XPAT returns 0 hits).

Comment 13

[Erik Wikström]

(In reply to comment #12)
> It's not wise to exclusively rely on XPAT support because when XPAT isn't
> supported on an NNTP server, the user is left scratching their head wondering
> why the search is broken.  Novices won't do protocol traces, so novices won't
> figure out that XPAT support is required server-side, and then be satisfied to
> restrict themselves to doing subject and sender searches only on NNTP servers
> that support XPAT.

Would it be possible to do both? I.e. try to search the server and if that fails  perhaps warn the user and do a serch on the local files?

> 
> Also, Thunderbird's use of using:
> 
> XPAT SUBJECT 1- <subject query>
> 
> only works correctly if the user is searching only for subjects only.  The
> default search in TB is "Subject or Sender" which requires *two* XPAT commands
> (at a minimum), one on the Subject header, and one on the From header, then
> merging the results from both XPAT commands.

It seems to me like the searches done by the quick-search is done on the local messages only (at least for sender & subject) whereas the searches made with the search-dialog depends on the server. (There is a server I can search using the quick-search but which returns no hits using the search dialog.)

Comment 14

[Joshua Cranmer [:jcranmer]]

Comment 4 seems to indicate that there are some servers which do not implement XPAT correctly, as opposed to not implementing XPAT at all (bug 98836). So this isn't quite a duplicate.

Comment 15

[Joshua Cranmer [:jcranmer]]

Does anyone see an example of a news server that doesn't support XPAT correctly? If not, I'll close this bug in two weeks.

Comment 16

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to last question. If you feel this change was made in error, please respond to this bug with your reasons why.