59681 - Newsgroup incorrectly appears to have unread messages

Status: RESOLVED FIXED

(MailNews Core :: Database, defect, P3)

Description

[Darin Fisher]

Often a newsgroup will appear in the sidebar to have unread messages when in
fact it does not.  Sometimes closing and reopening the newsgroup sub-tree (ie.
clicking on the triangle left of "news") will cause the newsgroup unread count
to refresh to the proper value.  In such cases, if I click on the newsgroup
that previously showed an incorrect unread count, the same incorrect unread
count will again appear.  This happens despite the fact that none of the news
messages are bold.

Noticing this on Linux [build 2000-11-07-09-MN6].

Comment 1

[R.K.Aa.]

*** Bug 60470 has been marked as a duplicate of this bug. ***

Comment 2

[Sebastian Fernandez]

I have also observed this on NT SP6, so it is not just a Linux problem.
Mozilla build 2000120504.

Comment 3

[Darin Fisher]

Yeah.. I too have seen it on WinNT sp5.  Marking OS:ALL.

Comment 4

[Asa Dotzler [:asa]]

I see this regularly.  If I'm in threaded mode, the top of the thread has the
unread icon but all messages in the thread are clearly read.  If I move to flat
mode then I see no messages (set to only show unread messages).  THis makes view
Unread in threaded mode pretty broken especially when you have a dozen threads
that claim they have unread items.

posibly realted to bug 42361 or more likely bug 32369 reassigning to
[e-mail] for triage or duping.

Comment 5

[Darin Fisher]

I also noticed that marking all read in threaded mode does not clear the unread
icon.

Comment 6

[Sebastian Fernandez]

I managed to fix this on my setup (Build 2001011204) by following a suggestion
from Ben Bucksch on n.p.m.mail-news:

<snip>
- Delete .msf files in your profile 
</snip>

After deleting all the .msf files in my profile, all my newsgroups and mail
folders report the correct number of unread messages.

Comment 7

[David :Bienvenu]

yes, the database and the newsrc file are out of sync - deleting the msf files
will fix this. The problem is to figure out why they're getting out of sync.

Comment 8

[Sebastian Fernandez]

Well, I don't know if this helps, but I've had this problem ever since I first
migrated my profile when N6 came out. I suspect migration may be the culprit.

Comment 9

[Sebastian Fernandez]

Never mind, the problem is back again, after 1 day of working correctly.

Comment 10

[Keyser Sose]

*** Bug 66747 has been marked as a duplicate of this bug. ***

Comment 11

[Mack Earnhardt]

I've been messing with this on and off for about a week and here's what I know
(or think I know) so far.

The reason unread counts in the sidebar change is due to moz using 3 different
sources for those numbers as its state changes. On opening mail/news, the
numbers come from panacea.dat in the profile. Since the server listing is open,
moz fetches the highwater for each newsgroup and compares it to newsrc to get
count #2. Finally, when you select a newsgroup, it fetches new headers and
updates the count based on msf thread header counts. Interesting, it does this
whether the newsgroup is displayed threaded or not.

You'd expect the initial panacea numbers to change depending on how long it's
been since you were in mail/news, but if they change when you select the
newsgroup that's a problem. As mentioned above, deleting the msf file is the
quickest workaround.

As to how this happen, I'll describe two methods but they both involve msf
updates and mailrc updates not being flushed to disk together.

Method #1 - this happens more frequently, at least to me
Open mail and read some messages in a newsgroup. If you switch to another
folder, don't come back to the original.  After a few minutes (5 minute cycle?)
newsrc will flush to disk.  Kill the mozilla process to simulate a crash.  You
now have threads showing more unread message than actually show as unread in the
pane. If you mark a few unread, you can get an unread count higher that the
total count. Imagine clicking N N N N N through all your message and going off
into the browser and you crash an hour later.  Grrrr...

Method #2:
Open a newsgroup and read some messages. Switch to another folder, then back to
the original. The original folder flushes to disk when you come _into_ it again.
Kill the process to simulate a crash. If you killed if before the newsrc cache
timed out and flushed, you now have threads showing unread counts less that the
number marked as unread. If you read (past tense) everything in the newsgroup it
looks very strange because the sidebar says several messages unread until you
select the newsgroup and it changes to zero unread and shows you a blank pane.

Comment 12

[TommyBee (No E-mail)]

It's not just a PC issue, either -- I'm experiencing the problem on Mac OS 9.1,
build 2001040608 and have been experiencing the problem for months of builds. 
It seems that the correct message status is lost if Mozilla crashes during a
session during which I've read messages.  My workaround lately has been to
immediately quit Mozilla right after I finish reading the new messages, then
relaunch Mozilla to do any other surfing -- it's an inconvenient way to do
things, but it works when I remember to do it.

I'm only a second year computer science major, so I don't know if this is
possible, but perhaps there might be some way to close the newsgroup files
immediately when the Mail / News window is closed?

Comment 13

[Håkan Waara]

Nominating to mozilla0.9 per Sebastian Fernandez request in npm.mail-news

Comment 14

[Sebastian Fernandez]

Should this bug be targeted to 0.9.1? 0.9.2? Do people consider this serious
enough to block 1.0? I know it annoys me, but I'm not sure if anyone else is
seeing it...

Comment 15

[Daniel Bratell]

Nominating for 0.9.1 and 0.9.2 (if 0.9.1 is too soon).

Comment 16

[jsight]

I definitely agree that this should block version 1.0 at least.

Comment 17

[Aleksey Nogin]

I believe this bug is a dup of #64476

Comment 18

[adam]

I would really like to see this bug resolved...

I think it is a definite blocker for 1.0

I would prefer it was fixed in .9.2

It's very consistently reproducible. It happens to me across platforms, builds
and completely new profiles. The synch is lost almost immediately on a fresh
install with a new profile.

While we've all dealt with worse bugs ; )  this is definetly visible and obvious
to even casual news users.

Comment 19

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 76166 has been marked as a duplicate of this bug. ***

Comment 20

[Aleksey Nogin]

Actually, both the number of unread messages and the total number of messages
would be wrong if .msf is stale.

Crossposted with bug #64476:

I was looking at what is happening in nsMsgNewsFolder and I observed the following:
1) nsMsgNewsFolder::UpdateSummaryFromNNTPInfo would set mNumUnreadMessages and
mNumTotalMessages to correct values.
2) nsMsgNewsFolder::UpdateSummaryTotals would call ReadDBFolderInfo that would
reset both of them to stale values.

On biff only (1) would happen, so the values would be correct (untill I try to
read the group, which will cause (1) and (2)).

Comment 21

[Aleksey Nogin]

I am starting to get an idea where things get unsynchronized. IMHO the problem
lies in nsNewsDatabase implementation. Namely, it keeps a field m_readSet (of type 
nsMsgKeySet *) and uses it to keep the information on whether a message is read
or not. However the global and thread read/unread counts are handled by an
underlying 
nsMsgDatabase which has its own idea of read/unread flags.

Although one can try to synchronize neswrc (which is used to produce m_readSet
with .msf (that is used by nsMsgDatabase) outside of nsNewsDatabase, IMHO the
best way to ensure that things never get unsynched is to keep them synched in
the nsNewsDatabase itself.

Because of that I beleieve that the best place for sync code would be in
nsNewsDatabase::SetReadSetStr - whenever some "outsider" changes the ReadSet we
should immediatelly propagate all changes to the underlying nsMsgDatabase.

P.S. Currently, nsNewsDatabase::SetReadSetStr does not have any synch code in it.

Comment 22

[Aleksey Nogin]

Attachment: Add some sync code to nsNewsDatabase::SetReadSetStr

Comment 23

[Aleksey Nogin]

I wrote some synchronization code that updated the DB based on the newsrc
information. I still need to write some code to synch thread counters (for bug
64476), but current patch already fixes this particular bug.

Comments, reviews?

Comment 24

[Aleksey Nogin]

I attached an updated patch - see bug 64476, attachment 38926 [details] [diff] [review].

Comment 25

[hirata masakazu]

*** Bug 89763 has been marked as a duplicate of this bug. ***

Comment 26

[basic]

*** Bug 90810 has been marked as a duplicate of this bug. ***

Comment 27

[Keyser Sose]

*** Bug 80806 has been marked as a duplicate of this bug. ***

Comment 28

[Keyser Sose]

*** Bug 83855 has been marked as a duplicate of this bug. ***

Comment 29

[Michael Hendy (Hendikins)]

*** Bug 91187 has been marked as a duplicate of this bug. ***

Comment 30

[dave.close]

Text from bug 91187: Using Ctrl-Shift-C to mark all messages in a newsgroup as
read clears the count of total messages in the "mail folders" pane and changes
all message headers to non-bold. However, many message topics in the message
list pane still show a non-zero count in the "unread" column. This means that,
when you return to the newsgroup a few days later, you cannot easily distinguish
which message threads are actually new and unread.

I believe this report is slightly different than the meat of this bug. It is not
the count of unread messages in a newsgroup that is wrong in this case, it is
the count of unread messages in a thread. However, it may have the same root cause.

Comment 31

[Håkan Waara]

Dave: work is going on on that issue (thread "looks" unread), see bug
64476. FYI, a partial fix went in just today for that one.

Comment 32

[Aleksey Nogin]

Yes, bug 91187 is a duplicate of bug 64476, not of this one.

Comment 33

[[e-mail] (gone - use [e-mail] instead)]

I've corrected bug 91187 to be a DUP of 64476, sorry about that (too quick to
verify the wrong bug).

Comment 34

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 87066 has been marked as a duplicate of this bug. ***

Comment 35

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 91660 has been marked as a duplicate of this bug. ***

Comment 36

[James Ross]

This bug has definatly got my vote!

I'm using build 2001090408 on Windows 2000. I just read my unread newsgroup
message, then when I click the folders I get (usually, if I have read all
messages in that folder) 1 unread and some weird number for the total (always <=
real total). The numbers are then all corrected when Moz does it's 'check for
new messages'.

Comment 37

[James Ross]

I found out something that might be related to this (using build 2001091303):

On some of my newsgroups Mozilla insists there are unread messages when it
automatically updates the message counts, but if I go to the group there turn
out to be no new messages at all. However, after looking at the .rc file for the
news server (while Mozilla wasn't running, of course), I decided to
'concatinate' the messages numbers in the file. By this I mean that some groups
had lots of different message ID ranges (e.g. 1-256,258-476,496-720), which I
replaced with just one range (e.g. 1-720). Now Mozilla does not seems to get the
'phantom' unread messages.

I suspect, but can't prove, that the number of phantom unread messages was equal
to the number of message IDs not included in the ranges.

Comment 38

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 99963 has been marked as a duplicate of this bug. ***

Comment 39

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 108245 has been marked as a duplicate of this bug. ***

Comment 40

[Sebastian Fernandez]

I haven't seen this bug in a while. Can anyone else out there with a fresh
profile still reproduce it?

Comment 41

[Gordon Ross]

I'm using Moz 0.9.5 and I still get it, although not as regularly as I used to.

Comment 42

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 122679 has been marked as a duplicate of this bug. ***

Comment 43

[Gabor Liptak]

It seems that the autoupdate ("check for new messages") processes the count
properly mostly, but it goes wrong when one actually selects the group for
reading (?)

Comment 44

[Gabor Liptak]

using 2002020803 I looked into Additional Comment #37 From James Ross 2001-09-20
05:50 

This latest code seems to clean up the ranges, if a newsgroup is fully read,
than it mostly seems to contain one range (1-max). I did see one newsgroup
(fully read) with multiple ranges (could be attributed to cancels?), and I made
it into only one range. When the server is opened in the newsreader (and
autoupdated), the count is correctly zero, and it changes to unread count when
the newsgroup is selected for reading (altough no messages are displayed).

Comment 45

[Vidar Haarr (not reading bugmail)]

I'm using 2002021915, and this issue has bugged me since I started using Mozilla
for MailNews.

Could we get this for 0.9.9 ?

Comment 46

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 126302 has been marked as a duplicate of this bug. ***

Comment 47

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 128005 has been marked as a duplicate of this bug. ***

Comment 48

[Gabor Liptak]

I do not know if changes were made for this, but it seems that lately using
"Mark Newsgroup Read" seems to set the unread messages correctly to 0 and it
stays 0 for a while.

Comment 49

[Vadim Berezniker]

I agree. Using "Mark newsgroup as read" fixes the bad counts. 
(It didn't used to before). 
But this is still only a workaround. The real problem still needs to be solved.

Comment 50

[Greg K.]

Confirmed using FizzillaCFM/2002031005 (0.9.9). Setting Platform=All.

Comment 51

[Vadim Berezniker]

*** Bug 136859 has been marked as a duplicate of this bug. ***

Comment 52

[Boris 'pi' Piwinger]

*** Bug 143628 has been marked as a duplicate of this bug. ***

Comment 53

[Boris 'pi' Piwinger]

*** Bug 147441 has been marked as a duplicate of this bug. ***

Comment 54

[Aleksey Nogin]

A sync patch was checked in in bug 64476

*** This bug has been marked as a duplicate of 64476 ***

Comment 55

[[e-mail] (gone - use [e-mail] instead)]

Verified dup. 

Comment 56

[Aleksey Nogin]

Unfortunatelly, there still appear to be some scenarious when unread counts
start to mismatch - often (some) cancelled messages would be counted as unread
in newsrc, but not counted in db!

Should I file a new bug or reopen this one?

Comment 57

[David :Bienvenu]

Neither, please use bug 79130

Comment 58

[Boris 'pi' Piwinger]

This also happens for new messages in ignored threads, which is not bug 79130.
Annoying.

pi

Comment 59

[David :Bienvenu]

see bug 147110 for the ignored threads issue.

Comment 60

[Boris 'pi' Piwinger]

Bug 147110 is the opposite. I claim that unread messages are counted for the
group where they actually are in ignored threads.

pi

Comment 61

[David :Bienvenu]

It seems more likely that one of you is wrong - I doubt that both could be true.

Comment 62

[ketil]

On Mozilla 2002051319, it looks as if unread messages are counted for the group when they actually are in ignored threads, at least on my computer. 

Comment 63

[Aleksey Nogin]

katil, as comment #59 said, see bug 147110 for the ignored threads issue.

Comment 64

[Boris 'pi' Piwinger]

*** Bug 166583 has been marked as a duplicate of this bug. ***