305918 - Duplicated news headers from a second nntp connection when pressing Get Mail/Msgs - need to prevent second server connection

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: NNTP, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Build ID: Trunk 2005-08-25 on Windows XP SeaMonkey and Thunderbird.

Summary: Duplicated headers when pressing Get Mail/Msgs from a second connection.

1. Subscribe to alt.tv.alias (~4439 articles)
2. Choose "Download all headers", click Download
3. Let it start downloading
4. Go to your Inbox
5. Come back to alt.tv.alias

Now, this is the *extra* step I left out of part (B):

6. Click the Get Mail button.

Now you'll have an extra copy of each header.

Comment 1

[Wayne Mery (:wsmwk)]

still see this problem?

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Screenshot of 2007-06-23 Thunderbird trunk showing the bug

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Although for a different group (I didn't yet have nttp:5 logging enabled), http://sedonner.tripod.com/alt.english.usage.zip contains my .msf file, and should show the same issue.  In the case of alt.english.usage, I pressed GetMsg while downloading the 704th message header.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

http://sedonner.tripod.com/mozilla-log.txt has the logfile of--unfortunately--another session, but one that exhibits the issue as well (same group).  You can very clearly see in it the two NNTP networking threads; the second being when I pressed Get Mail while the first thread/connection was already downloading the postings.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Question to clarify:
"Order Received" for newsgroup looks to be "article number" of the post.
What number is displayed in "Order Received" column of the duplicated articles?

By the way, I could observe duplicated article number too, using mozilla.support.thunderbird of news.mozilla.org, with release build of Tb 2.0.0.4(MS Win XP), but re-creation procedure in my environment was different from yours - No GetMsgs in my case.
 1. Subscribe a news group, and click the newsgroup
    => dialog => reply download of all articles (about 40,000)
 2. While downloading by step 1, click other account or mail folder,
    and click again the newsgroup on which download is in progress
    => dialog(second) => reply download of all articles (about 40,000)
    => "count display update at status bar" for each download
 3. While downloading by step1 & step2, click other account or mail folder,
    and click again the newsgroup on which download is in progress
    => dialog(third) => reply download of all articles (about 40,000)
    => "count display update in status bar" for each download
 4. Count in folder pane after completion of download(s) : 120,000 = 40,000*3
 5. Duplicate article numbers in "Order Received" column.
 6. "Rebuild Index" corrects wrong message count.
No locking while downloading when newsgroup?

Comment 6

[WADA:World Anti-bad-Duping Agency]

Changing to Component=MailNews:Database because it doesn't seem to be network related issue. 

Comment 7

[David :Bienvenu]

it's not a database issue - two nntp urls are running at the same time, both adding headers to the database. It's a mail backend issue, or possibly a networking : nntp issue.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Changing to Product=MailNews:Backend according to Comment #7, because I believe Networking itself works very well(it works very well, even if multiple downloads are requested for a newsgroup on a news server).

Comment 9

[David :Bienvenu]

I don't think the nntp code should allow you to have two connections going to the same newsgroup.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #9)
> I don't think the nntp code should allow you to have two connections going to the same newsgroup.

I don't think so.
I believe current network logic and database logic works very well, and concurrent news server access to single news server are executed very well, even if concurrent download to single target database, except duplicated message count and duplicated entries in ".msf". This indicates that current database logic handles multiple download requests to single database very well.
Serialization or locking can be achieved at two places, "Server access" and "Database access", and "where to serialize" depends on implementation, and implementation can freely choose it. But I believe that best place to serialize is "database", instead of "server access".
Please consider next case.
 - Parallel download from single server to multiple database 
Although this is not currently supported, this is possible, if account of same server/port/user-id will be supported. In this case, serialization by server name or address is not reasonable. Serialization by database is reasonable. And if "serialization by download target database", this bug's problem won't occur.

Comment 11

[David :Bienvenu]

there's a one to one correspondence between target database and a particular newsgroup on a particular server. It doesn't really make sense to put this code in the database, architecturally. It doesn't care about news or nntp, and why would you want to have two connections to a news server, both downloading headers for the same newsgroup? It's redundant, and only adds unneeded server load.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #11)
> why would you want to have two connections to a news server, 
> both downloading headers for the same newsgroup?
> It's redundant, and only adds unneeded server load.
I see. I'm not opposite to "serialization in server access", because I believe "where to serialize" always depends on implementation.
Change to appropriate Component, please.
By the way, please note that I always respect your design/decision, except for "default of Global Inbox" ;-)

Comment 13

[Wayne Mery (:wsmwk)]

(In reply to comment #11)
> there's a one to one correspondence between target database and a particular
> newsgroup on a particular server. It doesn't really make sense to put this code
> in the database, architecturally. It doesn't care about news or nntp, and why
> would you want to have two connections to a news server, both downloading
> headers for the same newsgroup? It's redundant, and only adds unneeded server
> load.

Comment 14

[David :Bienvenu]

I did just put code in the db to prevent this, because it turns out that the db has some unhappy threading if identical messages (with the same message key) are added to a thread. But we still shouldn't allow a newsgroup to get two updates going at the same time.

Comment 15

[Wayne Mery (:wsmwk)]

nntp has new code for a couple years now. I don't see evidence of this issue.