140555 - Offline: Newsgroups downloaded with "Download/Sync now" are dl'ing double amounts of messages

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Backend, defect)

Description

[Adam Macznik]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0rc1) Gecko/20020417
BuildID:    2002041711 - 1.0RC1

When downloading newsgroups for offline with "File->Offline->Download/Sync now"
option I see Mozilla downloads twice as much messages, as there are on the
newsgroup.

Reproducible: Always
Steps to Reproduce:
1.Subscribe to a newsgroup, for example netscape.public.mozilla.test.
2.Choose Sync from Offline menu.
3.Select that newsgroup in the window opened with "Select.." button.
4.Click OK
5.See the newsgroup headers downloaded. The number of new messages shown in the
left pane is, say, 10.

Actual Results:  The text appears in the status line: "Downloading messages
1-<nn> in <newsgroup>". In the example, the number <nn> would be 20! And Mozilla
really downloads 20 messages (I have a modem connection and I can see it takes
time do download messages).

Expected Results:  The text should say "Downloading messages 1-10 in
netscape.public.mozilla.test" and Mozilla should download only as many messages
as there are, not twice as much.

I did some further investigation and found that the file
<...profile>/News/netscape.public.mozilla.test really has 20 messages, but first
11 of them is the same message! Also noticed that during download, the first
message gets downloaded immediately (the icon to the left of the title turns
grey), then nothing changes till we reach the count of 11 on the status line -
an then next icons start to turn grey.

When I select "Download now" button in newsgroup properties, only 10 messages
would be downloaded. [Well, sometimes I see that it downloads for example "1-14
messages", but then first 4 would be "downloaded very fast" - so I presume it
just skips some headers for some reason. So I think it's not important].

This bug is rather annoying on large newsgroups and slow or breaking
connections. And it consumes disk space.

Comment 1

[grylchan]

Using commercial branch
Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.0rc1) Gecko/20020426

I am seeing something here. 

Seems like there bunch of 'bug's reporter is reporting here.
I'll try to address each one.
  -Sometimes the count in left pane of messenger is off
   numerous bugs on that like bug 108650
  -When I did a download/sync now on newgroup the count in left
   pane said 35 and it actually did download 35 mesgs
  -I tried downloading a newsgroup with lots of mesgs
    -you get the prompt that it contains over 1000 mesgs do you
     want to download 500. I say yes 500 only
    -when I download the headers, the left pane says there are only 223
     but in status update at the bottom of messenger it says
      'downloading 330 of 455'
     I'm sure this a known bug.  Stephend know the bug number for this?

  -I also noticed reporter is correct when you go into online store
   folder for a newsgroup that it does indeed have multiple copies
   of the first header. That's a bug.
   And if you do properties->download now button on newsgroup that
   you DON'T get the mutlitple headers of the newsgroup.

So i am confirming the bug on multiple headers in the online store
file for a newsgroup that is downloaded via download/sync now.

Comment 2

[Adam Macznik]

Sorry if I wasn't specific enough.
Yes, I meant this bug to be about multiple messages in local store. But I think
it's only a result, that's why I wrote that long description.

Re: -When I did a download/sync now on newgroup the count in left
     pane said 35 and it actually did download 35 mesgs
Yes, only 35 msgs are displayed, but still in the status line there would be
"Downloading articles 1-70 in news.group" and the local file in
<profile>/News/<newsserver>/news.group would have 70 messages. When I copy this
file to Mail/Local Folders, 70 msgs are displayed, first 36 of them the same
[when it is recognized as a local folder after Mozilla restarts].

Comment 3

[David :Bienvenu]

yes, I'm seeing this too. It's very odd - I believe this code works by doing a
search for messages not downloaded for offline use, and adding each search
result to the list of hdrs to download. So I suspect something has gone wrong
with search, but I'll look into it.

Comment 4

[David :Bienvenu]

Attachment: proposed fix

problem was we were passing in the keys in m_keysToDownload to
nsNewsDownloader::DownloadArticles(), which turned around and inserted them
into m_keysToDownload! So we doubled the articles. The fix is to simply pass in
null, and nsNewsDownloader::DownloadArticles will use the values already set in
m_keysToDownload. I would put in code to clear out m_keysToDownload in
nsNewsDownloader::DownloadArticles, but that would clear out the pointer we're
trying to pass in :-(

Comment 5

[(not reading, please use [e-mail] instead)]

Comment on attachment 81633 [details] [diff] [review]
proposed fix

sr=sspitzer

Comment 6

[David :Bienvenu]

fixed on trunk. I'm going to nominate for the release simply because the fix is
so simple and because it will save half the disk space used by offline news.

Comment 7

[David :Bienvenu]

*** Bug 143298 has been marked as a duplicate of this bug. ***

Comment 8

[grylchan]

Using commercial trunk builds:
2002-05-21-08-trunk/ nt 4.0
2002-05-21-07-trunk linux 2.2
2002-05-21-03-trunk/ mac 9.2.2
2002-05-21-08-trunk/ mac 10.1.4

Verified when you do a download/sync now on a newsgroup
that the offline store file (../News/newsgroupfilename)
does indeed only contain 1 copy of each of the news mesgs
that were downloaded. There are no 'doubles' of any messages in
the offline store file. And the number of messages in the store
file matched number of messages listed in folder pane of messenger.

Also checked using 'download now button' in the properties window of
a newsgroup and that to has correct number of headers in offline store folder.

marking as verified on trunk.

If gets approved for branch and lands on branch, will then reverify this bug.

Comment 9

[David :Bienvenu]

*** Bug 147532 has been marked as a duplicate of this bug. ***

Comment 10

[David :Bienvenu]

*** Bug 149587 has been marked as a duplicate of this bug. ***