Last Comment Bug 298737 - newsgroups periodically lose unread count for messages whose headers haven't been downloaded
: newsgroups periodically lose unread count for messages whose headers haven't ...
Status: RESOLVED FIXED
: fixed1.8.1.1
Product: MailNews Core
Classification: Components
Component: Backend (show other bugs)
: Trunk
: All All
: -- normal with 10 votes (vote)
: ---
Assigned To: David :Bienvenu
:
Mentors:
: 301057 325299 335843 355896 (view as bug list)
Depends on:
Blocks: 363966 364063
  Show dependency treegraph
 
Reported: 2005-06-24 21:36 PDT by Ryan Hill
Modified: 2008-07-31 01:21 PDT (History)
19 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
work in progress (2.25 KB, patch)
2006-12-08 15:43 PST, David :Bienvenu
mscott: superreview+
Details | Diff | Splinter Review
fix counts issue (3.88 KB, patch)
2006-12-20 19:01 PST, David :Bienvenu
mscott: superreview+
Details | Diff | Splinter Review
adjust unread counts for message headers we don't have (2.94 KB, patch)
2006-12-21 11:52 PST, David :Bienvenu
no flags Details | Diff | Splinter Review
adjust unread counts for message headers we don't have, but just once for any given header (3.40 KB, patch)
2006-12-21 12:55 PST, David :Bienvenu
mscott: superreview+
Details | Diff | Splinter Review

Description Ryan Hill 2005-06-24 21:36:42 PDT
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8b2) Gecko/20050623 Firefox/1.0+
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8b2) Gecko/20050623 Firefox/1.0+

last produced with TB 1.0+ 20050623.  first noticed over a month before this.

after retrieving message headers from the news server, often after a short time
(15 minutes or so) the unread messages count for each news group other than the
one selected will vanish.  choosing a group will cause the previous count to
reappear or update if any new messages are available.  this seems to happen
sporadically and so far i don't have a reliable way to reproduce it.

i don't have TB set to retrieve new news messages periodically, but i do for
mail and RSS.  it's a possibility that this timer might be involved as it's the
only thing i know set to check every 10-15min.  i have news set up to never
delete messages and i don't work offline so the bodies don't get retrieved until
i click to view them.

i'll try turning off the check for new mail timers and download message bodies
beforehand to see if makes a difference.

Reproducible: Sometimes

Steps to Reproduce:
Comment 1 Ryan Hill 2005-06-24 22:10:14 PDT
nope, neither did.  but i think i noticed a bit of a pattern.

- have x number of newsgroups, each with unread messages.
- check the server for new messages.
- there are new messages retrieved for some but not all of the groups.
- a short time later the groups that did happen to have new messages available
will keep their unread count while groups that didn't get reset to 0.
Comment 2 Minh Tran-Le 2005-06-26 09:22:58 PDT
Had it happen to me on 2 different machines with slightly different
configuration but same news server (news.supernews.com).

I think that it is something the news message expiring system. I have it set to:
- Delete messages more than 45 days.
- Only message body less than 30 days old.
Comment 3 Ed Buffey 2005-08-18 10:25:13 PDT
This bug also affects Windows. It's not consistent but it happens enough that
I've been watching this bug. I only recently realized that it's marked for Linux.
Comment 4 Karsten Düsterloh 2005-08-18 14:11:36 PDT
*** Bug 301057 has been marked as a duplicate of this bug. ***
Comment 5 Ruediger Lahl 2005-08-19 05:52:22 PDT
(In reply to comment #0)

> after retrieving message headers from the news server, often after a short time
> (15 minutes or so) the unread messages count for each news group other than the
> one selected will vanish.

Hey, I'm not alone :-) See https://bugzilla.mozilla.org/show_bug.cgi?id=301057

Meanwhile I can say that exactly every 5minutes some newsgroups are affected,
until all unread message counts vanished.

> choosing a group will cause the previous count to
> reappear or update if any new messages are available.

Or you simply fold in and out the server in the folderpane.

> this seems to happen
> sporadically and so far i don't have a reliable way to reproduce it.

Here every time/5min and absolutly reproduceable.

> i don't have TB set to retrieve new news messages periodically, but i do for
> mail and RSS.  it's a possibility that this timer might be involved as it's the
> only thing i know set to check every 10-15min.

I searched 'about:config' for events that starts every 5minutes. I changed all
found items to an other time, but the Bug still works in his 5min timeperiode.

I also do not find any errors in smtp/nntp-logs.
Comment 6 Steve Wendt 2005-10-29 12:06:00 PDT
Seen in Seamonkey on Win32 and OS/2.
Comment 7 Ruediger Lahl 2005-10-29 12:40:23 PDT
(In reply to comment #6)
> Seen in Seamonkey on Win32 and OS/2.

Thanks for reporting.
Is that bug new for you? Did you change the version/build? Buildnumber?
Comment 8 Steve Wendt 2005-10-29 12:50:05 PDT
(In reply to comment #7)
> Is that bug new for you? Did you change the version/build? Buildnumber?

No, it's been around for quite a while (1.0a release and earlier, haven't tried later builds).  I just now bothered to look for it being listed in Bugzilla.  :)
Comment 9 Ryan Hill 2005-10-29 13:22:56 PDT
i think this occurs for newsgroups that have new messages available but the headers of those msgs haven't been downloaded.  once you actually click on the newsgroup the headers are grabbed and it seems to work (for that group).
Comment 10 Ruediger Lahl 2005-10-29 14:06:13 PDT
(In reply to comment #9)
> i think this occurs for newsgroups that have new messages available but the
> headers of those msgs haven't been downloaded.

Yes.

> once you actually click on the
> newsgroup the headers are grabbed and it seems to work (for that group).

The newsgroup in the focus is never affected and postings that once was read and after that marked as unread again are also not affected.

For me the most interesting thing is, the 5minutes-intervall the bug do his work. Three persons verifying the bug now. Should we set blocking to '?' now?
Comment 11 Ryan Hill 2005-10-29 21:43:29 PDT
nah, without at least an idea of what the cause is i doubt this qualifies, especially since it's so close to release time.  it's annoying but it's been around since at least 0.8.  a little longer won't hurt. ;)
Comment 12 Ryan Hill 2005-11-22 18:58:14 PST
updated summary to be more relevant.

it might also be worth mentioning that if there are unread messages that _do_ have their headers downloaded (eg. during a previous session) the unread count will revert to that number.  
Comment 13 Ryan Hill 2005-12-24 02:52:01 PST
okay, this is getting even worse on trunk.  all you have to do now is hover the cursor over a newsgroup title for a couple seconds and watch the unread message count disappear.  can anyone confirm if this happens on the branch?

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20051223 Thunderbird/1.6a1 ID:2005122306

req blocking.
Comment 14 Ruediger Lahl 2005-12-24 14:53:24 PST
(In reply to comment #13)

> okay, this is getting even worse on trunk.  all you have to do now is hover the
> cursor over a newsgroup title for a couple seconds and watch the unread message
> count disappear.  can anyone confirm if this happens on the branch?

You mean, I only have to put the mouse over a newsgroupname in the folderpane and after some seconds the newsgroup lost her unread message count?

No, I cant confirm that, but the unread message count always disappears periodical after exact five minutes. Every five minutes three or four NGs are affected until all NGs are thru.

> req blocking.

Yes ACK.
Comment 15 Ruediger Lahl 2005-12-25 13:56:55 PST
(In reply to comment #13)

> Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20051223
> Thunderbird/1.6a1 ID:2005122306

Okay, now I tested with Thunderbirds actual nightly-build, and now I can confirm that. I made a little .avi from my screen to show what happend here.

The DivX-avi as a zip-file(only 49KB) is here: http://lahls.de/public/counts_gone.zip
Comment 16 idgreenwald 2006-01-20 18:01:01 PST
Seen in Thunderbird on Win XP SP2
Comment 17 Ruediger Lahl 2006-03-03 07:33:53 PST
(In reply to comment #0)
> after retrieving message headers from the news server, often after a short time
> (15 minutes or so) the unread messages count for each news group other than the
> one selected will vanish.

Meanwhile the bug had made it into Branch. Some other people in de.comm.software.mozilla.mailnews confirmed the bug for SM1.0

Someone should look for it now.
Comment 18 Chris Ilias [:cilias] 2006-07-10 19:50:15 PDT
*** Bug 335843 has been marked as a duplicate of this bug. ***
Comment 19 David E. Ross 2006-09-03 08:33:51 PDT
Given the frequency of complaints about this in <news://news.mozilla.org:119/mozilla.support.thunderbird>, could the priority of this be raised?  
Comment 20 Magnus Melin 2006-10-08 07:07:14 PDT
*** Bug 355896 has been marked as a duplicate of this bug. ***
Comment 21 David E. Ross 2006-10-08 07:50:55 PDT
I saw this yesterday in an extreme form.  I subscribe to 18 newsgroups on one server.  I opened the server.  Before Thunderbird finished working its way down the list, refreshing the unread counts on each newsgroup, the counts that had already been refreshed started going blank with the newsgroup names changing from bold to normal.  
Comment 22 David :Bienvenu 2006-12-08 15:43:04 PST
Created attachment 248023 [details] [diff] [review]
work in progress

The problem is basically that hovering over a message causes us to open the db, which causes us to only count the headers we've downloaded. I have a patch that basically borrows a concept from the imap code: pending counts, i.e., counts for headers we know about but haven't downloaded yet.

This doesn't totally fix the problem, but I think it improves things quite a bit - if anyone wants to try to run with this patch, please give it a shot.
Comment 23 Ruediger Lahl 2006-12-08 16:13:03 PST
(In reply to comment #22)
> The problem is basically that hovering over a message

For me its not necessary to hovering the groups. The unread message counts begins to disappears if I wait exactly 5 minutes. Then the first 4 or 5 groups lost their counts. After another 5 minutes the next groups are following, and so on until all groups lost their counts. Reproducible: always since more than a year :-(

> basically borrows a concept from the imap code: pending counts, i.e., counts
> for headers we know about but haven't downloaded yet.
> 
> This doesn't totally fix the problem, but I think it improves things quite a
> bit - if anyone wants to try to run with this patch, please give it a shot.

Sorry, I can not test the patch cause I use Tinderbox, but I like to test a compiled win32-build with this patch included.
Comment 24 Magnus Melin 2006-12-10 05:24:54 PST
Tried the patch, it certainly fixes bug 318792! 
(Can't say about this bug though, since i think I haven't run in to it.)
Comment 25 David :Bienvenu 2006-12-10 07:51:17 PST
Comment on attachment 248023 [details] [diff] [review]
work in progress

the patch definitely seems like an improvement
Comment 26 David :Bienvenu 2006-12-10 13:29:18 PST
this fix is landed - I think there are still issues, but I think the main issue is fixed. 
Comment 27 Mike Cowperthwaite 2006-12-11 17:47:23 PST
(In reply to comment #26)
> this fix is landed - I think there are still issues, but I think the main
> issue is fixed.

Is this reasonable?  The original bug was filed before the New Message Folder Popup was even implemented, and (as already noted) bug 318792 was open already for that popup-related problem.

Also, I just ran across bug 325299, which seems to be a dupe of this original bug.
Comment 28 David :Bienvenu 2006-12-11 17:51:43 PST
this fix has nothing to do with new mail alerts, and the original bug was just a side effect of new mail alerts - the problem arose whenever the db for the newsgroup was opened - we would throw away all the counts we got from the server, and just use the counts of headers we'd downloaded. Hovering over a newsgroup was causing us to open the db, and exposing the bug. As did having retention settings run, which also causes the db to get opened.
Comment 29 Ruediger Lahl 2006-12-12 07:37:35 PST
(In reply to comment #26)

> this fix is landed - I think there are still issues, but I think the main issue
> is fixed. 

I would like to test the patch, but their are no new 'WINNT 5.2 sea-win32-tbox Clbr VM-release'-builds since nearly three days in Tinderbox. Isn't it possible to fix that VM?
Comment 30 David :Bienvenu 2006-12-12 07:44:28 PST
see bug 363292 for the tinderbox issues.
Comment 31 Ruediger Lahl 2006-12-12 13:27:03 PST
(In reply to comment #30)
> see bug 363292 for the tinderbox issues.

Thanks David. Meanwhile it looks like they set up another VM which build a nightly.
Comment 32 David :Bienvenu 2006-12-12 14:57:06 PST
*** Bug 325299 has been marked as a duplicate of this bug. ***
Comment 33 Ruediger Lahl 2006-12-19 12:10:56 PST
(In reply to comment #26)
> this fix is landed - I think there are still issues, but I think the main issue
> is fixed. 

Newsgroups don't lose unread counts anymore. But I think, their is a regression.
I opened a new bug[1], because I can not exactly say, that this patch cause the bug.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=363966
Comment 34 David :Bienvenu 2006-12-20 19:01:05 PST
Created attachment 249328 [details] [diff] [review]
fix counts issue

If there are missing headers on the server (e.g., a message is cancelled), we were always showing that message as unread. So instead of clearing the pending counts when we start opening the folder, clear them when we've finished downloading headers for the folder. This fixes the problem for me...
Comment 35 David :Bienvenu 2006-12-21 11:52:12 PST
Created attachment 249369 [details] [diff] [review]
adjust unread counts for message headers we don't have

this goes through the unread messages and checks if we have the headers for all of them - if we don't, it marks them read. 

This works for me - the one thing I'm unhappy about is that it does this every time you click on a newsgroup, and if you keep thousands of unread messages, we'll  check them all every time. Ideally, we should keep track of what we've already checked and not check them again. I'll see if I can do that.

This should fix a really long time frustrating issue for news readers...
Comment 36 David :Bienvenu 2006-12-21 12:55:56 PST
Created attachment 249378 [details] [diff] [review]
adjust unread counts for message headers we don't have, but just once for any given header

this keeps track of the range of headers we've checked previously. 

If an article expires after we've downloaded the header, I don't think we care, since the user can mark it read anyway.
Comment 37 Scott MacGregor 2006-12-21 20:05:58 PST
Comment on attachment 249328 [details] [diff] [review]
fix counts issue

I'm assuming we want both this patch and the other one  I just reviewed for this bug...
Comment 38 David :Bienvenu 2006-12-22 05:57:42 PST
I've landed those two fixes on the trunk and branch as well.
Comment 39 Ruediger Lahl 2006-12-22 11:34:33 PST
(In reply to comment #36)
> Created an attachment (id=249378) [edit]
> adjust unread counts for message headers we don't have, but just once for any
> given header

Thanks David. The patches works perfect for me.
Comment 40 David E. Ross 2007-03-04 08:03:11 PST
In what release of Thunderbird was this fixed?  I seem to still see this problem in Thunderbird version 1.5.0.10 (20070221) with WindowsXP Sp2.    
Comment 41 Steve Wendt 2007-03-04 11:42:59 PST
The fix is in the Gecko 1.8.1 branch , not the 1.8.0 branch.  That means Thunderbird 2.x and Seamonkey 1.1.x.

Note You need to log in before you can comment on or make changes to this bug.