Threads with an expired (or cancelled) message will show the unread thread indicator, even when all are read, and expired/cancelled message causes incorrect unread count of newsgroup

ASSIGNED
Assigned to

Status

MailNews Core
Networking: NNTP
ASSIGNED
16 years ago
a month ago

People

(Reporter: stephend@netscape.com (gone - use stephen.donner@gmail.com instead), Assigned: jcranmer)

Tracking

(Blocks: 4 bugs)

Dependency tree / graph
Bug Flags:
wanted-thunderbird3 +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [has workaround, see bug 294754 comment #3])

Attachments

(5 attachments)

Observed with 2001050608 on Windows 2000.

Summary: Threads with an expired message will show the unread thread indicator,
even when all are read.

Steps to Reproduce:

1.  Visit a newsgroup which has a thread that contains unread postings, but has
an expired message.
2.  Expand the thread, and read all of the postings.
3.  You can even try Message | Mark | Thread as Read, but this doesn't work, either.

Expected Results:

When you read all of the messages in a thread, the thread icon shouldn't have a
green "unread" indicator on it.

Actual Results:

We show the message status as being "read" in the columns to the right, but the
icon doesn't update to reflect that we've read the whole thread.
QA Contact: esther → stephend
nice catch, stephen

top of my head guess:  I bet when we mark the "expired" message as read, we are
unable to find thread it belongs to and so we fail to adjust the unread count on
the thread.

but I'd have to debug to find out.

cc'ing bienvenu.

Comment 2

16 years ago
Or, we're marking it read without going through the db (e.g., just poking the
hdr flag), and thus not adjusting the thread unread count.
bienvenu's scenario sounds more likely.

accepting.
Status: NEW → ASSIGNED
weird.

at first, I was unable to reproduce this.

but then (using Mark | Thread | Read?) I was able to get into a bogus state
where the unread count on the thread was clearly wrong.

stephend, when you see this problem, what's the unread count for the thread? 
(it's in the "Unread" column, second from the right in the thread pane)
related weird note:  when I get into this bad state, my unread count is negative
(-1 in my case) and that causes the green icon in the thread column.

nsMsgDBView.cpp, line 4287, we ask the thread for the unread count which is -1.

but it's an unsigned int, so -1 is really a big int, which is > 0 so we show unread.

Investigating...
The unread count for that thread indicates 1, so I'm assuming that would be the
expired message.
can you attach a screen shot so I can see if it is what I'm seeing?

Created attachment 33417 [details]
Screenshot of thread
ok, I think we're seeing the same thing.

start up, find a thread with a read, expired article.  click on the green dot to
mark that expired article as "unread".  the unread count for the thread doesn't
change.

after the first click, it works fine.  but the initial change doesn't work so
the unread count on the thread is off by 1.

I'll work on that scenario first.
Yeah, sorry, in fact I had played with that (toggling the status icon) but
forgot to include that in my original filing.  Thanks for clarifying that, Seth.
the first time you mark an expired message as unread, it's not in the read set,
so we think it is already unread.

we've got code that will bail out if you are trying to mark an unread message as
unread (or a read message as read).  see nsMsgDatabase::MarkHdrRead()

still working on this...  I think nsMsgDatabase::MarkHdrRead() is correct.

I need to investigate why the expired message isn't in the read set.
weird, looking in my news.rc file, I have something like this:

1-5000,5773,5774

5773 is my thread root, 5774 is my expired thread child.

I would have expected to see 5773-5774.

perhaps that's at the root of the problem.

(the reason it would work on subsequent clicks is I think the unread set code
caches the last thing.)

Comment 13

16 years ago
taking, I'm looking at these issues.
Assignee: sspitzer → bienvenu
Status: ASSIGNED → NEW

Comment 14

16 years ago
Created attachment 43371 [details] [diff] [review]
db part of fix

Comment 15

16 years ago
I changed the underlying db methods to relax the restriction on not marking msgs
read that are already read, in the case that the hdr flags are set differently
than the newsrc flags. (the !! stuff is because you need to make sure booleans
have been normalized to 1 and 0 before comparing them for equality). I also
cleaned up a lot of unused stuff in nsNewsDatabase.cpp. Cc'ing Navin for review
and asking Seth for sr. (I'm not really sure this patch fixes this exact
problem, but it is good for other stuff too - this problem might be fixed by
other changes as well.).
Status: NEW → ASSIGNED
sr=sspitzer

Comment 17

16 years ago
r=naving

Comment 18

16 years ago
Created attachment 44253 [details] [diff] [review]
view diffs to sync up flags, and use the newsrc for displaying read/unread status for news msgs

Comment 19

16 years ago
Navin and Seth, can I get a couple more reviews on the last patch? It just makes
sure we use the newsrc flag setting instead of the db setting, and moves some
common code into a function. The common code also makes sure the db flags
correspond to the newsrc flag when we open the view.

Comment 20

16 years ago
r=naving
sr=sspitzer
David, do you have anymore cleanup work on this bug?

1.65 bienvenu%netscape.com Aug 2 06:43   more work on syncing news read state 
between db and newsrc r=naving, sr=sspitzer 79130 

I'll take a look at the client and see if my problem is fixed, but should this 
be marked fixed for me to verify now?

Updated

16 years ago
Blocks: 106267

Comment 23

16 years ago
Stephen, you can look into it. I've checked in various fixes to sync up the
state but I don't know if will fix this particular case of the threads with
expired messages.
This still occurs with the 2002-01-03-03 build on Windows 2000.

Here's the behavior remaining:

If an expired article exists in a thread, and you read all of the messages in
the thread, we still display the unread thread icon.

If you run the list-id? url on the newsgroup, and come back to the thread, we
still display the unread thread icon.

I don't think this is major, especially since it's news, and happens mostly with
expired articles.
Created attachment 63410 [details]
Screenshot of thread with the expired posting
Created attachment 63411 [details]
Screenshot of thread after the expired posting has been removed

Comment 27

15 years ago
Still see this with BuildID 2002061018 on Red Hat Linux 7.2 (but for total
counts, have not checked thread counts). It appears that the newsrc consideres
these messages to be unread, while db does not.
Summary: Threads with an expired message will show the unread thread indicator, even when all are read. → Threads with an expired (or cancelled) message will show the unread thread indicator, even when all are read.

Comment 28

15 years ago
I sure hope this is the active bug for this.

I think I have a little insight into this bug.  My symptom is that a newgroup
will default to some number (say 9).  If more messages actually come in, the
reported number of unread messages in the sidebar will be 9 plus the number of
new messages.  I was given a "fix" for the bug, which was deleting the file
specific to the newsgroup and restarting Moz.  It almost worked.  Now, the
newsgroup title is just bold normally.  When I select it, it now displays  a 1
in parenthesis, even though there are no new messages. 
Also, it seems that it is directly related to the number of posts I've made on
that newsgroup.
I only get this bug for newsgroups that I've posted on (I read a lot, and post
little).
I've been reading a lot of posts for this bug, but I've never heard it mentioned
that this bug interferes with the "next" button.  It won't move past the thread
with the false positive in it.

Comment 29

15 years ago
This bug also occurs in build 2002082608 (1.1 final) running on Windows XP. 

Updated

13 years ago
Blocks: 236849
Product: MailNews → Core

Comment 30

12 years ago
*** Bug 160963 has been marked as a duplicate of this bug. ***

Comment 31

11 years ago
I've frequently experienced a bug resembling this, but I'm not sure if this is it.  I certainly haven't pinpointed the cause to expired or cancelled messages.

Sometimes a thread will appear as having unread messages (both the green arrow and the underlining) even though there are none.  They even clutter the Threads with Unread and Watched Threads with Unread views.  In some cases, expanding the thread and then collapsing it clears the status; in others, the thread is stuck forever in this damaged state.  The only way to get rid of it seems to be to either unsubscribe and resubscribe or to delete the .msf file, thereby losing all watched/ignored threads and downloaded message bodies.

Indeed, I am experiencing it on a news server that never expires messages - so obviously it isn't expired messages causing the problem, but could still be something to do with cancelled messages.  And so waiting for the whole thread to expire and it stops getting in the way isn't an option.

Maybe having a sample .msf file that shows the problem would help.  I'll try to supply one next time the problem comes up.

Comment 32

11 years ago
I'm experiencing the same problem described by Stewart Gordon, with additional caveat that subscribe/resubscribe does not seem to work every time.

Comment 33

10 years ago
I'm still experiencing this problem with version version 1.5.0.9 (20061220).

I synchronize the .rc file across several computers, and then read the news from all these computers.  unread messages are correctly highlighted.  So if I show only unread messages I actually have only the unread messages.  But if I show Threads with Unread, I get also threads where all messages are read, in particular if on that computer the first message of the thread was never opened (it was read on another computer).

Comment 34

10 years ago
I've been seeing this problem for about as long as I've been using Mozilla products, and I still see it in TB 2.0.

What appears to be happening is that the .rc file for the affected news server is not being correctly updated.  I can correct the problem for a newsgroup that should be all read but shows messages as unread in the folder pane by manually editing the .rc file outside of the application, and changing the line to

<newsgroup name> 1-<lastnum>

When I go back into teh app, teh counts will be correct.

What I haven't been able to isolate is why random lines in .rc files aren't being correctly updated.
______
Dennis
Duplicate of this bug: 370234
Duplicate of this bug: 166980
Duplicate of this bug: 239153
Blocks: 71728
JFY.
Accordigng to meta Bug 71728, Bug 24592 & Bug 34406 seem to be oldest ones for download dialog, and Bug 41457 seems to be oldest one for newsgroup count display.
JFY.
Bug 108650 seems to be oldest one for different count before open newsgroup from count after open newsgroup. 
Adding "incorrect count" in summary to reflect symptom of DUPed Bug 160963 by comment #30 for ease of search.
David(bug owner), please correct it, if this bug is only for "unread indicator for already downloaded article which was canceled or expired after download". 
Summary: Threads with an expired (or cancelled) message will show the unread thread indicator, even when all are read. → Threads with an expired (or cancelled) message will show the unread thread indicator, even when all are read, and expired/cancelled message causes incorrect unread count of newsgroup

Comment 41

10 years ago
Are you sure the problem is in the .rc file?
As far as I can see, those files are updated correctly, in fact, if I use "show unread messages" I actually see only the messages that are not read.

The problem is when I use "view thread with unread", and I think the problem is due to the fact that thunderbird uses the .msf and .dat files instead of actually inspecting all messages in a thread (e.g., compare the message id against the ids of the .rc file).

This happens to me since I read newsgroups from the same server on different machines and I always use the same .rc files (I keep them synchronized), while the .msf and .dat files are different on the different machines.

Thus, if I mark all messages of a thread on a machine, copy the .rc file onto another machine and reopen the newsgroup, the same thread is shown even if all of its messages are read.

I don't know about the internals of thunderbird on this subject, but I suspect that this is the problem...

you may also reproduce this behavior by simply remove the .msf and .dat file of a newsgroup, then reopen thunderbird, reopen that newsgroup, choose "view threads with unread" and see this wrong behavior...

This bug makes thunderbird very hardly usable for reading newsgroup, and it's quite astonishing it hasn't been fixed yet...

if, at least, someone could provide information about the part of code responsible for dealing with threads with unread one could try to fix the problem...

thanks in advance
(In reply to comment #41)
> Are you sure the problem is in the .rc file?
[...]

I am (or at least that's where "part" of the problem lies): messages canceled before having been read cannot be "marked as read" in the rc file other than by hand-editing, and they are counted as "unread" in the newsgroup in the left pane; see bug 294754 comment #3.

Comment 43

10 years ago
(In reply to comment #42)
> (In reply to comment #41)
> > Are you sure the problem is in the .rc file?
> [...]
> 
> I am (or at least that's where "part" of the problem lies): messages canceled
> before having been read cannot be "marked as read" in the rc file other than by
> hand-editing, and they are counted as "unread" in the newsgroup in the left
> pane; see bug 294754 comment #3.

I don't know whether it's where the actual problem is, but it's certainly the visible evidence.  Newsgroups are displayed in the folder pane in the order in which they appear in the .rc file, and read/unread counts are stored in the line for each individual group.  (I've re-arranged the order in which broups for a server are listed by editing the .rc file for that server -- it would be nice if there were a GUI interface to do that...)

Manually correcting the appropriate lines in the .rc fixes the innacurate read/unread display in the folder pane.  I have no idea where in the TB code that functionality resides, or whether it's in the code that actually updates the rc file, or the code that keeps the info used to do so.

Unlike Tony, I haven't tried to cancel messages.  My problems occur simply in reading.  The read counts are not always correctly updated.

I wouldn't go so far as to say it makes TB unusable for reading news -- it's more of an annoyance -- but since I only use TB as a newsreader, and handle email elsewhere*, it's a *big* annoyance.

* I use GMail as my primary email account, and prefer the web interface.  I don't get email as POP mail, though it's nice that TB 2 adds GMail POP support out of the box. (It would be nicer still if it did not default to trying to download mail as POP as soon as you set it up.  You might not want to do it right then, or at all.  The setup process for GMail should mention that is is setting things up for POP delivery, and let you specify when it should be done.)
______
Dennis

(In reply to comment #43)
[...]
> Unlike Tony, I haven't tried to cancel messages. [...]

I haven't either, but I've had messages canceled on me (by moderators or whomever) before I had come around to reading them, causing the problem.
Duplicate of this bug: 294754

Comment 46

10 years ago
Sorry if I insist on this: I think the .rc file is updated correctly, it is the view "threads with unread" that is not correct.

Here's how to reproduce the problem:

1. subscribe to a new newsgroup (e.g., alt.test)
2. open it and download the headers (even a part of it, if they're too many)
3. mark all as read
4. close thunderbird
5. remove the .dat and the .msf files
6. re-open the newsgroup
7. select "thread with unread" and all the threads will be shown even if all the messages are read

As I said, this happens to me since I read newsgroups from the same server on different machines and I always use the same .rc files (I keep them synchronized), while the .msf and .dat files are different on the different machines.

Thus, if I mark all messages of a thread on a machine, copy the .rc file onto
another machine and reopen the newsgroup, the same thread is shown even if all
of its messages are read.

The steps above is only to simulate this situation...
(In reply to comment #46)
> Sorry if I insist on this: I think the .rc file is updated correctly, it is the
> view "threads with unread" that is not correct.

Sorry if I insist on this: I never use the "Threads with unread" view, I only use "View All", yet at times there are newsgroups which appear bolded and with a number in the left pane; then when I select that NG, there is no unread msg after all, except as soon as I move away the bold & count reappear. The only way I've found  to make the problem disappear is by hand-editing the newsrc as I originally posted at bug 294754 comment #3. The way I see it, the "deleted" or "expired" message reappears as unread when the counts are refreshed and cannot be marked permanently as read from within Thunderbird.

Comment 48

10 years ago
(In reply to comment #46)
> Sorry if I insist on this: I think the .rc file is updated correctly, it is 
> the view "threads with unread" that is not correct.

Have you actually *looked* at the .rc file when this happens?

The .rc file is *where* the information on what is and is not read displayed in the folder pane is stored.

If you bring up the .rc file in an editor on a group where you've marked everything as read and posts still show as unread, you'll see the problem.

Here's a sample .rc file from my system, for newsgroups on the Baen server:

baen.1632Tech: 1-155504,155512,155516-155518,155526,155534,155543
baen.administrivia: 1-15915
baen.bar: 1-61263,61268-61270,61275,61277
baen.books: 1-17530
baen.buships: 1-28405,28420,28422,28423,28428,28430,28432-28434,28437
baen.classicsf: 1-1293
baen.EBookReader: 1-1836
baen.faq: 1-91
baen.FutureTech: 1-14769
baen.honorverse: 1-39405
baen.space: 1-4431

This: "baen.space: 1-4431" indicates all messages are read.

This: "baen.bar: 1-61263,61268-61270,61275,61277" indicates that messages 1-61263 are read, 61264-61267 are *not* read, etc.

Look at the .rc file for a group which shows an unread message count in the folder pane but all messages are in fact read, and you'll see that the line in the .rc file for that group is wrong. 

When I see that problem, I can correct it by editing the .rc file to include the correct range of read messages.  When I restart TB, the problem is gone.

What I *can't* reproduce is *why* a particular entry in the .rc file isn't properly updated.  It happens randomly, occurs on any of the six news servers I have configured, and doesn't affect all groups on the server. 
______
Dennis
(In reply to comment #48)
[...]
> This: "baen.bar: 1-61263,61268-61270,61275,61277" indicates that messages
> 1-61263 are read, 61264-61267 are *not* read, etc.
[...]

They aren't read, and, in the case concerned by this bug, cannot be read because they don't exist anymore. I think there are in theory two possible fixes for this bug:

a) Leave the rc as it is, but don't count messages that don't exist on the server anymore when computing the number of unreads for a group; or
b) when it is found that a message doesn't exist anymore, mark it as read in the rc file so the mailer will remember that that message isn't "unread" (in the sense of being available for reading).

I think (b) would be easier to implement, since (a) wouldn't leave a memory of which "holes" in the list correspond to messages that cannot be read anymore.

Comment 50

10 years ago
(In reply to comment #49)
> (In reply to comment #48)
> [...]
> > This: "baen.bar: 1-61263,61268-61270,61275,61277" indicates that messages
> > 1-61263 are read, 61264-61267 are *not* read, etc.
> [...]
> 
> They aren't read, and, in the case concerned by this bug, cannot be read
> because they don't exist anymore. I think there are in theory two possible
> fixes for this bug:
> 
> a) Leave the rc as it is, but don't count messages that don't exist on the
> server anymore when computing the number of unreads for a group; or
> b) when it is found that a message doesn't exist anymore, mark it as read in
> the rc file so the mailer will remember that that message isn't "unread" (in
> the sense of being available for reading).
> 
> I think (b) would be easier to implement, since (a) wouldn't leave a memory of
> which "holes" in the list correspond to messages that cannot be read anymore.

I agree that (b) would be easier to implement.  But the basic issue is that the .rc file is wrong.  For the purposes of this discussion, it doesn't matter whether the messages exist and *are* all read, or no longer exist on the server and can't be read.  The .rc file is showing unread messages when it should not. 
______
Dennis

Comment 51

10 years ago
(In reply to comment #48)
> (In reply to comment #46)
> > Sorry if I insist on this: I think the .rc file is updated correctly, it is 
> > the view "threads with unread" that is not correct.
> 
> Have you actually *looked* at the .rc file when this happens?
> 
> The .rc file is *where* the information on what is and is not read displayed in
> the folder pane is stored.
> 

the .rc file is good, in fact if I switch to the view "show only unread messages" I actually see no message (since all are read): the problem is only with the view "show threads with unread"...

probably I'm posting in the wrong bug?
Duplicate of this bug: 365819
Whiteboard: [has workaround, see bug 294754 comment #3]

Comment 53

9 years ago
See comment #7 in issue 365819.
(Assignee)

Updated

9 years ago
Duplicate of this bug: 307873
(Assignee)

Updated

9 years ago
Duplicate of this bug: 391173
Please do proposal (b) in comment 49!  
I think this would fix SO MANY problems.  Maybe even bug 400526.

Updated

9 years ago
Blocks: 438257

Updated

9 years ago
Duplicate of this bug: 356537
(Assignee)

Comment 58

9 years ago
(In reply to comment #49)
> a) Leave the rc as it is, but don't count messages that don't exist on the
> server anymore when computing the number of unreads for a group; or
> b) when it is found that a message doesn't exist anymore, mark it as read in
> the rc file so the mailer will remember that that message isn't "unread" (in
> the sense of being available for reading).

Recalling a bit from memory, the newsrc file is written by dumping the key set. I can see the value of remembering "holes" on a bigger scale, since it would be nice to know that we haven't tried looking at a block of articles. But on the other hand, if we download articles 1-1000 and don't see article 334, it's safe to assume that we won't ever see 334.

Questions to ponder on:
* What criteria should we use to determine which missing messages should be glossed over in the key set?
** (Tentative response: XOVER-time. But what about interrupted downloads?)
* How should Mark All As Read work?
** (Tentative response: reset the newsrc to <oldest>-<latest>. But how does this interact with Get Next N Messages?)
* What if we do find 334 later on?
** (Tentative response: Mark it as unread and reset read set information)

Reassigning to myself (and resetting QA). Hope bienvenu doesn't mind :-).
Assignee: bienvenu → Pidgeot18
Status: ASSIGNED → NEW
QA Contact: stephend → networking.news
(Assignee)

Comment 59

9 years ago
Hit commit too early (and need to accept bug anyways)...

In any case, it looks like b) is the way I'm going to go, although it's a b-with-comments.
Status: NEW → ASSIGNED
Flags: wanted-thunderbird3.0a2?

Updated

9 years ago
No longer blocks: 438257

Updated

9 years ago
Depends on: 438257

Updated

9 years ago
Blocks: 438257
No longer depends on: 438257

Updated

9 years ago
No longer blocks: 438257

Comment 60

9 years ago
The only portion of the summary I can verify is the incorrect message count resulting from message cancels.  This I have seen on news.annexcafe.com, a server that honors user news cancel requests.  The other case involving that server was cancels of spam, which have been curtailed by the server going to all users requiring authentication. Then My low traffic group was effected often.  Seemed like message numbers were getting recycled. Possibly involved some server reindex. Them a cleanup of the *.rc file was not enough and clearing the effected *,msf was needed.
(Assignee)

Updated

9 years ago
Duplicate of this bug: 24592
(Assignee)

Updated

9 years ago
Duplicate of this bug: 41457
(Assignee)

Updated

9 years ago
Duplicate of this bug: 185972
Flags: wanted-thunderbird3.0a2? → wanted-thunderbird3?
Product: Core → MailNews Core
(Assignee)

Updated

8 years ago
Duplicate of this bug: 475705
(Assignee)

Updated

8 years ago
Duplicate of this bug: 476293
Flags: wanted-thunderbird3? → wanted-thunderbird3+
Blocks: 265465
Duplicate of this bug: 466825

Updated

7 years ago
Duplicate of this bug: 446228
You need to log in before you can comment on or make changes to this bug.