43278 - Crossposts (same Message-ID) not marked as read in other newsgroups

Status: NEW

(MailNews Core :: Networking: NNTP, defect)

Description

[Henrik Gemal]

If you read a newsposting that are send to multiple newsgroup, the newsposting 
should be marked read in all the newsgroups when reading the newsposting in one 
newsgroup.
If you fx crosspost to netscape.public.mozilla.license 
netscape.public.mozilla.mail-news fx:
Newsgroups: netscape.public.mozilla.license, netscape.public.mozilla.mail-news

and you've subscribed to both of the newsgroups and the first read the 
newsposting in the "license" newsgroups. When you then go to the "mail-news" 
newsgroup the posting should already be marked read.

Just like in Netscape 4.73

It must be something with the "Message-ID:" header. Isn't that unique?

Not working in Build 2000062020

Comment 1

[(not reading, please use [e-mail] instead)]

ui, moving to m18.

accepting.

Comment 2

[David :Bienvenu]

nominating for nsbeta3

Comment 3

[scottputterman]

Mail Triage is marking [nsbeta3-]

Comment 4

[laurel]

*** Bug 56238 has been marked as a duplicate of this bug. ***

Comment 5

[Ben Bucksch (:BenB)]

Changing SUMMARY to contain "crosspost".

Comment 6

[[e-mail] (gone - use [e-mail] instead)]

Mass moving all NEWS bugs from esther to myself.

Comment 7

[Alan S. Jones]

This gets really anoying when my brain is sleeping and I read the same mozilla
news posting 3 times in 3 groups

Comment 8

[Eric Vaandering (no email)]

*** Bug 90661 has been marked as a duplicate of this bug. ***

Comment 9

[Eric Vaandering (no email)]

Adding message ID to subject to pick up more searches

Comment 10

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 92026 has been marked as a duplicate of this bug. ***

Comment 11

[scottputterman]

adding nsenterprise-

Comment 12

[Håkan Waara]

Gonna have a try at this... (don't expect anything just yet).

Comment 13

[(not reading, please use [e-mail] instead)]

hwaara, what I described (over aim) is not going to work.

I'll investigate the right way to fix this and let you know.

Comment 14

[(not reading, please use [e-mail] instead)]

here's the change from what we talked about over aim:

when we mark a message as read, you need to get and handle the Xref header.

here's an example from a cross post:

Xref: secnews.netscape.com netscape.public.mozilla.seamonkey:6274
netscape.public.mozilla.qa.general:866 netscape.public.mozilla.mail-news:15135

for each group, see if we are subscribed to that group on the server using
nsINntpIncomingServer::ContainsNewsgroups().  if we are subscribed, you'll need
to get the nsIMsgNewsFolder and from that get the nsINewsDatabase, and use
getReadSet() to get the read set, and then you'll add the key to the read set.

I'll go find the 4.x code that parses the Xref header so you can port it over.

Comment 15

[(not reading, please use [e-mail] instead)]

Attachment: 4.x code that parses the Xref header.

Comment 16

[Håkan Waara]

Seth, I assume I have to edit nsIMsgDatabase and nsIMsgHdr to have a xref member
too (it has for other common headers...)?

Comment 17

[(not reading, please use [e-mail] instead)]

talking it over with bienvenu, here's how we suggest doing this:

look at the nsIMsgHeaderSink interface.  that interface is implemented in 
msgHdrViewOverlay.js.

extend the interface so that you can get headers back out of it,

wstring retrieveHeader(in string name);

implement that in msgHdrViewOverlay.js on the messageHeaderSink object, it 
should be a simple matter of returning currentHeaderData[name].headerValue

then, in nsNNTPProtocol, in MarkCurrentMsgRead(), get the header sink from the 
m_msgWindow, call retrieveHeader() on the "xref" header.  (do lower case, the 
currentHeaderData lookup is by lowercase header name.

be wary of the fix for http://bugzilla.mozilla.org/show_bug.cgi?id=104519, when 
mscott fixes that, we'll need to make sure xref is still being passed through 
to the header sink.  even though we aren't displaying it, we'll need it for 
this to work.

Comment 18

[Håkan Waara]

Working on this now, looks good so far.

One big change is that I will parse the "Newsgroups" header instead of Xref.

Reasons: * It's a required header according to the standards
         * It doesn't contain all the unnecessary, extra stuff the
            Xref header does

Comment 19

[Håkan Waara]

My previous comment assumes there is nothing we want to fetch from the 
newsgroup header, except for the names of the newsgroups the message was cross-
posted to.

Comment 20

[(not reading, please use [e-mail] instead)]

> Reasons: * It's a required header according to the standards
>
>         * It doesn't contain all the unnecessary, extra stuff the
>           Xref header does

that extra stuff is the article number, which you need or you can't implement
this feature.

if you haven't seen the message in the other newsgroups, you have no way of
marking it as read, unless you have the article number.  (with the article
number, you can set it as read getting the read set.

if all you've got is the newsgroups header, you can't mark cross posts as read.

Comment 21

[Håkan Waara]

Thanks for clarifying!

Comment 22

[(not reading, please use [e-mail] instead)]

cc mscott, for his feedback to the proposed change to nsIMsgHeaderSink.

hold off on hacking this until he approves the approach.

he knows nsIMsgHeaderSink and msgHdrViewOverlay.js best.

Comment 23

[Håkan Waara]

FYI, I am already working on the xref parser. Getting the xref header via
nsIMsgHeaderSink works like a charm.

Comment 24

[Håkan Waara]

Won't work on this under existing feature-freeze.

Comment 25

[Phil "clef" Sweeney]

It is important to also carry across the kill-thread status to crossposted
newsgroups too.  Will this be catered for?

Would appreciate this bug getting fixed..

Comment 26

[laurel]

*** Bug 128498 has been marked as a duplicate of this bug. ***

Comment 27

[Alfonso Martinez]

*** Bug 129473 has been marked as a duplicate of this bug. ***

Comment 28

[Jens Müller (:tessarakt)]

Phil (#25): Maybe you should file this as a new RFE bug, as it is a seperate
feature.

Comment 29

[R.K.Aa.]

*** Bug 160285 has been marked as a duplicate of this bug. ***

Comment 30

[Frank Schönheit]

Attachment: proof of concept

This is a "proof of concept" in a sense that it implements the requested
functionality, but has drawbacks which hinder me to denote it as "patch".
Perhaps they are not serious, don't know - personally, I am willing to take
them :). I'm heavily interested in any opinions about this.

I had to leave the path sketched above for various reasons:

1. Using the nsIMsgHeaderSink from within nsNNTPProtocol did not work reliably.
It seems that at the very moment MarkCurrentMsgRead is called, the header sink
is not yet filled with the current header data, but still contains the data of
the previously selected header.

2. The header sink may be filled with some headers only (depending on the
View/Headers setting), so even if we could reliably use it, it may be that it
simply does not contain the xref-header line we're interested in, just because
the user chose to _not_ show it.

3. Even if we could use the xref from the header sink, to me it seems that this
would be the wrong way. Having the NNTPProtocol asking the header sink, which
is dedicated to a nice formatting of header lines, for a specific header line,
just to do some crosspost-marking, seems ... strange. Sounds a little bit like
misusing the header sink to me, sorry :)

4. Adding a message key to the ReadSet of a nsINewsDatabase does not completely
what is wanted (in my opinion). When I used it, then in the tree pane, the
statistics about the group where the message had been crossposted was not
updated until I explicitly selected this group. This is somewhat ugly, as it
leads to groups which pretend to contain n unread messages, and when you select
them, they say "just kidding, they're already read".
I did not find a way to synchronize the news database state with it's read set
(well, I did not look _too_ hard :).

5. From my pre-mozilla Mail/News-client, I know a nice additional functionality
(I do not know if Netscape 4.x did it this way, and if this is part of the
original request): It did respect crossposts not only when an x-posted messages
was selected (and thus implicitly switched to "read"), but for _every_ change
of the "read" state of an x-posted message.
This means no matter which message was actually selected, if the user clicked
onto the "read" flag of any other message in the folder, then the new "read"
state was propagated to all other folders containing this message (no matter if
the message had been marked as read or unread).
But, hooking into MarkCurrentMsgRead would not do this, as this seems to be
used (only) when implicitly switching a message to read (not when switching to
unread, and not when switching a non-current message).

I played around with it a little bit, and here is what the attached changes do:

First, I think I need to cache the xref-header in the news database. Given 5.
above, especially given the need (hmm - the wish :) to propagate the read
status even for non-current messages, it seems that the only way is to have the
xref header cached. So I introduced a "xref" column for news databases
(nsNewsDatabase.*).
The main drawback of this is that in order to make this work, the msf files for
news folders need to be deleted and rebuilt, and of course increase in size ...


Additionally, I use the nsMsgNewsFolder to handle the changes in the "read"
status of a message. The NewsFolder already is a KeyChange listener at it's
database (at least it's base class), so it only needs to override OnKeyChange
to get notified of all changes in the database. The folder has access to it's
NNTPIncomingServer, and thus can parse the xref-header, extract the groups,
locate them in the account, and handle the message therein.

Yet more, the code does not use the ReadSet of the database because of the
above-mentioned syncing problems. Instead, it retrieves the header directly,
and marks it as (un)read. This seems to be propagated to all interested parties
immediately, especially the tree view is updated.

Last, nsNNTPNewsgroupList.cpp was changed to not ignore the xref-header anymore
(it previously did), but forward it to the MsgHdr it is currently initializing.
Thus, this header can really be cached in the news database.


I am not sure if this is really the way to go. But it works :), and from my
(limited) knowledge of all the code in the mail/news area, it was the best way
I could think of.

Any opinions/flames welcome :)

Comment 31

[Frank Schönheit]

another drawback I "forgot" [1] to mention: The marking only works when the
folder where the messag has been x-posted to has already been visited, means the
header has already been downloaded for this folder.
This, of course, makes the proof-of-concept somewhat useless ...


[1] well, I noticed it too late, I admit my tests were not extensive enough :(

Comment 32

[Stewart Gordon]

What algorithms do newsreaders tend to use for marking crossposts as read, for
that matter?

Two approaches that would generally work are quite simple:

1.(a) When a crossposted message header is downloaded, and the crosspost is to
other subscribed 'groups, check whether it has been read in those 'groups and
update the status accordingly.  If the header hasn't been downloaded in any
other 'groups, leave as unread.
(b) When a crossposted message has been read, and the crosspost is to other
subscribed 'groups, mark it read in the other 'groups.

2. Store a global index of message IDs and the read/unread status of each.  Of
course, don't forget garbage collection.

A further improvement would be (at lesat as an option) to propagate watch/ignore
for crossposted messages in the same way.

Comment 33

[Gábor Lipták]

any progress is being made on this please?

Comment 34

[Gábor Lipták]

any progress is being made on this please?

Comment 35

[Straxus]

Attachment: Duplicate message handling code from Gnus 5.10.2

This is duplicate suppression code from the Gnus newsreader. It's in Lisp, and
my Lisp is a little rusty, but it appears that the messageID is added to a list
of messageIDs which is then scanned at some other point in the news viewing
process.

From the attached file:
;; This package tries to mark articles as read the second time the
;; user reads a copy.  This is useful if the server doesn't support
;; Xref properly, or if the user reads the same group from several
;; servers.

What would be required to convert this code to properly use the Mozilla
architecture? I can make this into a C++ equivalent without much fuss myself,
however any explanation of where the best place to put this fix into in the
current Mail/News architecture (or whether it's better to wait until the post
1.4 world) would be very greatly appreciated as I'm still trying to wrap my
brain around Mozilla's architecture and structure.

Comment 36

[Michael Lefevre]

Straxus - firstly, that code is GPL, so it can't be used, converted, or used as
a basis for Mozilla code, because then Mozilla would have to be GPL'd as well.

also, as the comment you quoted says, that code is a fallback - "useful if the
server doesn't support Xref properly". Virtually all servers do support Xref, so
we should use Xref to implement this in the first case (I guess one could also
implement something message-id based as a fallback, but that's not important).

It's nice that you want to help, but it's really not worth trying to help with
coding things unless you have some understanding of how the Mozilla project code
works. Thanks.

Comment 37

[Ben Bucksch (:BenB)]

> It's nice that you want to help, but it's really not worth trying to help
> with coding things unless you have some understanding of how the Mozilla
> project code works. Thanks.

Hey. That sounds like a polite "you have no clue, go away". Everybody who
started had no idea how the Mozilla code works, we all had to get into it, and
that's what he's trying, see his last sentence.

Comment 38

[Straxus]

Mike:

First, I'm aware that Gnus is GPL'ed. Perhaps I'm not exactly fully
understanding of how such licensing works, however Mozilla *is* partially GPL
(see http://www.mozilla.org/MPL/ - Mozilla is/will be tri-licensed - MPL, GPL,
and LGPL).

Second, I'm not some clueless twit that knows a few pieces of C++ and wants to
help like a good little 12-year-old. A lot of other people that you may be used
to seeing replying to bugs may be like that, but I am not. I just graduated from
Honours Computer Science at the University of Waterloo, and I've read over the
architecture documentation for Mozilla. I'm currently very busy looking for a
job and don't have all of my time to devote to learning an architecture that's
about to go away anyways, which is why I was looking for pointers. I figured
those with the knowledge of the architecture could save me hunting time in the
interest of a faster fix.

I've added the Gnus code because someone asked how other readers do it. We
already have an example of Xref handling from NS 4, so I also added an example
of a backup algorithm from another newsreader. Anyways, being able to handle
servers that munge the Xref is a good thing from my point of view as I've always
been one to make robust code when possible.

As for persistence across sessions, it appears that some sort of external file
would be necessary for containing messageIDs if the messageID route were the one
to be pursued. I believe that using messageIDs offers some advantages over just
using Xrefs. For one, I like the fact that a message can be marked as read
across multiple news accounts, whereas I don't think that'd be possible with
Xrefs. I also like that it provides compatibility for servers with broken Xrefs,
as it adds robustness to the reader and heads off a possibly common bug
complaint before it arrives. However, the obvious tradeoff is speed. Every time
we enter a newsgroup, we will have to check the headers of every one of the new
and unread messages against our list of messageIDs to see what needs to be
marked read. As well, we need to choose an arbitrary default number of
messageIDs to keep in this file, and probably eliminate them in a FIFO manner as
new entries are added. In my opinion, 10,000 seems like a good number for this
and should handle most scenarios. And, if people want to change it, it'll most
likely be a pref.

For a solution that goes the Xref route, is Comment 17 in this bug still
relevant? It was made more than a year and a half ago, and I'm doubtful that the
architecture is still similar, so clarification would be nice.

Seth, how do you want to go on this one? You're the module owner, so you get to
choose the preferable route.

Comment 39

[Ben Bucksch (:BenB)]

> learning an architecture that's about to go away anyways

If you mean Messenger -> Thunderbird, Thunderbird is essentially Messenger with
the XUL and startup changed somewhat (to my understanding), I don't think the
core mail/news code changes at all during that change.

> For a solution that goes the Xref route, is Comment 17 in this bug still
> relevant? It was made more than a year and a half ago, and I'm doubtful
> that the architecture is still similar

I don't think the arch changed that much in the last time. Just look at the code
he pointed at and you'll see, if it's still relevant.

Comment 40

[Stewart Gordon]

I find it ridiculous that this isn't part of GNKSA (see bug 12699).  Indeed, I
find it ridiculous that GNKSA doesn't even stipulate a concept of read/unread
distinction.

Comment 41

[Frank Schönheit]

Attachment: proof of concept II (working, but unpolished)

This patch now works for about 3 months in different Mozilla builds (the diff
file is for 1.5), didn't notice any problems so far.
There are some open questions, and one problem which probably should be solved
differently (aka better), since in some situations, the patch is expensive in
terms of online traffic.
However, somebody might be interested in trying this, too ...

Comment 42

[Frank Schönheit]

forgot to mention: for this to work, you have to delete the .msf files for your
newsgroups, since they're now used to (also) store the Xref header.

Comment 43

[[e-mail]]

Comment on attachment 136045 [details] [diff] [review]
proof of concept II (working, but unpolished)

It says unpolished, so here's some polish ;-)

>+static PRBool isWhiteSpace( const PRUnichar c )
>+{
>+	return	( '\b' == c )
>+		||	( '\t' == c )
>+		||	( '\r' == c )
>+		||	( '\n' == c );
>+}
Doesn't seem worth it for the single use...

>+nsresult nsMsgNewsFolder::MarkReadInSiblingGroup(
>+	const nsCOMPtr< nsINntpIncomingServer >& _nntpServer,
Never, ever pass an nsCOMPtr around! Just use nsINntpIncomingServer *
>+	const nsCAutoString& _groupName,
Don't pass explicit string types around either. In this case you probably just
need a const char *
>+	const nsMsgKey _msgKey,
>+	PRBool _bRead )
Use "a" as a prefix i.e. aNntpServer, aGroupName, aMsgKey, aIsRead.

>+	NS_ASSERTION( _groupName.Length(), "nsMsgNewsFolder::MarkReadInSiblingGroup: invalid group name!");
Were you to use an nsString type the test would be IsEmpty() not Length().

>+		nsCOMPtr< nsIMsgNewsFolder > siblingNewsFolder;
>+		nsCOMPtr< nsIMsgFolder > siblingFolder;
>+		nsCOMPtr< nsIMsgDatabase > msgDatabase;
>+
>+		_nntpServer->FindGroup( _groupName.get(), getter_AddRefs( siblingNewsFolder ) );
>+		if ( siblingNewsFolder )
>+			siblingNewsFolder->QueryInterface( NS_GET_IID( nsIMsgFolder ), getter_AddRefs( siblingFolder ) );
nsIMsgNewsFolder doesn't inherit from nsIMsgFolder? I hope there's a good
reason for that :-/ Anyway, use siblingFolder =
do_QueryInterface(siblingNewsFolder);

>+				msgDatabase->QueryInterface( NS_GET_IID( nsINewsDatabase ), getter_AddRefs( newsDatabase ) );
Again, newsDatabase = do_QueryInterface(msgDatabase);

>+      msgKey = 0;
>+        msgKey = NS_STATIC_CAST( nsMsgKey, naturalLong );
nsMsgKey is just another typedef for unsigned long...

>+				HandleXRefHeader( xrefHeaderValue, MSG_FLAG_READ == ( aNewFlags & MSG_FLAG_READ ) );
!= 0 should do here, surely?

Also, all your whitespace is wrong, remember two spaces indent, no tab
characters, and if ( NS_FAILED( rv ) ) should be if (NS_FAILED(rv)) etc.

Comment 44

[Gábor Lipták]

Could some people provide feedback on this patch please?

I cannot build with Cygwin because of 

http://bugzilla.mozilla.org/show_bug.cgi?id=236635

Comment 45

[Juliano F. Ravasi]

I think that Frank Schönheit should polish his patch, as neil pointed and
helped, and then, set review and superreview flags to '?' on the new patch.

Comment 46

[David :Bienvenu]

const nsCOMPtr< nsINntpIncomingServer >& _nntpServer,

should just be nsINntpIncomingServer *nntpServer;

the method arg names don't correspond to our conventions, no leading '_'

it looks like there are tabs in the file - we use two space indent, no tabs

instead of passing around const nsString& _xrefHeader, just pass the const char
*, if you're not going to use the nsSTringness.

xref hdr should be a cstring - no reason to make it unicode that I can see.

I'm not sure about some of the other hacks in this - this should be a lot
simpler to do...

Comment 47

[Frank Schönheit]

I won't have time in the forseeable future to care for this patch, sorry :(

In any way, my "unpolished" was mainly about how the patch works, not about
syntactic issues (though I definately appreaciate Neil's and David's guidance
here!).

Trying to sketch what is my biggest concern with the approach I chose:

There are situations where I need to refetch the read/unread counts (which
are/can be displayed next to the newsgroup name) of a certain newsgroup.
However, it seems that the nsNNTPProtocol does not support retrieving those
number for only a *single group* - you can only do this for *all* groups of an
account.

Thus, I decided to retrieve not only the counts, but all the headers for this
group. This bears two problems:
- If you don't have a fast flat rate, you will most probably not appreciate this
- If you limited the headers to download per run, then you're bothered with the
"how many headers should I download" message box. The user experience of this is
somewhat weird: You mark a message in some.group.foo as read, and shortly after
this, you get this message box for some.group.bar.

These two points, IMO, render the patch useless for broader use - that's why I
named it "proof of concept" ...

A fix for this would be (so I suppose) to extend the nsNNTPProtocol so that it
can also retrieve counts for only a single group.

Comment 48

[David :Bienvenu]

Ok, thx, Frank. I'll try to drive this forward, or find someone else to do so. I
don't remember this being that complicated in 4.x, so I'll try to remember how I
did that.

Comment 49

[timeless]

bienvenu: there are news servers which carry chinese/korean named newsgroups,
are you sure normal strings are the way to go?

Comment 50

[David :Bienvenu]

those names have an ascii representation, I'm sure...

Comment 51

[Jungshik Shin]

(In reply to comment #49)
> bienvenu: there are news servers which carry chinese/korean named 

 I know there are Chinese servers that carries Chinese newsgroup names in
GB2312, but I have never seen any server with Korean newsgroup names, which
doesn't mean there isn't any but makes it rather unlikely. Anyway, those servers
are  broken.  

Comment 52

[Tapio Sokura]

What's the status of this feature request, is anyone working on it? I sure would
appreciate the feature, leaving crossposted articles as new in other groups
after they have been read in one group is the most annoying "bug" in the Mozilla
newsreader for me.

Comment 53

[Rupa Schomaker]

*** Bug 272021 has been marked as a duplicate of this bug. ***

Comment 54

[Kelley Cook]

Could this bug of not marking crossposts as read please be reassigned to someone
who is actually working on Thunderbird?

There are 171 Votes requests for this truly annoying missing feature which has
been part of every other newsreader for around 20 years.

Comment 55

[Juliano F. Ravasi]

I think that we already have everything to have this fixed on Mozilla. I'm
setting blocking1.8a6=? and blocking1.7.6=? so the lords can see if this can get
into mozilla. After this is fixed, I think it's just a step to commit the same
fix into thunderbird (although I don't know how to link this bug report to
thunderbird).

Comment 56

[Anil Daoud]

Is there anything preventing this bug from being fixed ? It has a fair amount of
votes and a patch...

Comment 57

[Stewart Gordon]

See comment 47.  I guess what we need is:

- someone to polish it who can figure out how
- someone to review it (whose job is this these days?)
- someone to check it in

But how do such fundamental issues take all these years to deal with in the
first place?

Comment 58

[Vesa Jääskeläinen]

(In reply to comment #57)
> But how do such fundamental issues take all these years to deal with in the
> first place?

I have a feeling that Spitzer is not anymore working with mozilla and after he 
have stopped it, components like this is still initially assigned to him. So no-
one responsible for component development doesn't even get a mail about issues. 
You can try to search bugs assigned to him and amaze how many those are there 
still open.

I don't even know where to report things like this, I'll hope someone sees this 
and can do better job on reporting component issue.

Comment 59

[Chris Woodfield]

(In reply to comment #58)
> (In reply to comment #57)
> > But how do such fundamental issues take all these years to deal with in the
> > first place?
> 
> I have a feeling that Spitzer is not anymore working with mozilla and after he 
> have stopped it, components like this is still initially assigned to him. So no-
> one responsible for component development doesn't even get a mail about issues. 
> You can try to search bugs assigned to him and amaze how many those are there 
> still open.
> 
> I don't even know where to report things like this, I'll hope someone sees this 
> and can do better job on reporting component issue.
> 

Maybe one could file a new bug referring to this one? A newly-filed bug would certainly be seen by 
active developers...

Comment 60

[coen]

> I have a feeling that Spitzer is not anymore working with mozilla

S. Spitzer is still the default bug-owner for 'Core Networking:News'.

The patch submitter should ask for review and superreview. Only the patch
submitter could do this.

Other sollution: Someone could download the patch and re-submit it. So the new
submitter is able to set the review and superreview flags.

Comment 61

[Michael Lefevre]

"The patch submitter should ask for review and superreview. Only the patch
submitter could do this."

Looking at the previous comments, the patch wasn't ready for review at the time.
14 months later, the patch will definitely need work before it can be reviewed.

There is no magic solution to make someone work on this (adding comments
definitely doesn't help). Either someone needs to hire a developer to do the
work, or we just need to wait for bienvenu or someone else to get around to it...

Comment 62

[Christian :Biesinger (don't email me, ping me on IRC)]

DO NOT file a new bug when a current one already exists. especially not if you
know about the existing one. that won't necessarily ensure that any developer
sees that bug anyway.

Comment 63

[Christian :Biesinger (don't email me, ping me on IRC)]

(and if it does, it will not guarantee that it will be fixed soon anyway)

Comment 64

[Felix Miata]

How can a 4xp, nsCatFood bug be an enhancement? Is there any other newsreader on
this planet that lacks this feature?

Comment 65

[Karsten Düsterloh]

This is no enhancement:
 Since every messages only exists once, it is read everywhere.
This is no major bug, either - it's just an inconvenience.

Comment 66

[David :Bienvenu]

I'm aware of this bug and would like to do something about it but there are
higher priority things on my list. If I can knock some of those off, I'll try to
drive this forward.

Comment 67

[Steve Sobol]

(In reply to comment #59)

> Maybe one could file a new bug referring to this one? A newly-filed bug would
certainly be seen by 
> active developers...

I just did, today, May 4th, 2005, and it got marked as a duplicate of this bug.
My post was bug #292870.

Comment 68

[Doug Wright]

*** Bug 292870 has been marked as a duplicate of this bug. ***

Comment 69

[Stephen J. Cowley]

(In reply to comment #66)
> I'm aware of this bug and would like to do something about it but there are
> higher priority things on my list. If I can knock some of those off, I'll try to
> drive this forward.

Thanks for all your work, but there are now 187 votes. Is there any chance of
this moving up the priority list? Or is it assumed that "news" is no longer
worth supporting?

Comment 70

[Karsten Düsterloh]

*** Bug 298644 has been marked as a duplicate of this bug. ***

Comment 71

[Patrick Dickey]

(In reply to comment #43)
> (From update of attachment 136045 [details] [diff] [review] [edit])
> It says unpolished, so here's some polish ;-)
> 
> >+static PRBool isWhiteSpace( const PRUnichar c )
> >+{
> >+	return	( '\b' == c )
> >+		||	( '\t' == c )
> >+		||	( '\r' == c )
> >+		||	( '\n' == c );
> >+}
> Doesn't seem worth it for the single use...
> 
> >+nsresult nsMsgNewsFolder::MarkReadInSiblingGroup(
> >+	const nsCOMPtr< nsINntpIncomingServer >& _nntpServer,
> Never, ever pass an nsCOMPtr around! Just use nsINntpIncomingServer *
> >+	const nsCAutoString& _groupName,
> Don't pass explicit string types around either. In this case you probably just
> need a const char *
> >+	const nsMsgKey _msgKey,
> >+	PRBool _bRead )
> Use "a" as a prefix i.e. aNntpServer, aGroupName, aMsgKey, aIsRead.
> 
> >+	NS_ASSERTION( _groupName.Length(),
"nsMsgNewsFolder::MarkReadInSiblingGroup: invalid group name!");
> Were you to use an nsString type the test would be IsEmpty() not Length().
> 
> >+		nsCOMPtr< nsIMsgNewsFolder > siblingNewsFolder;
> >+		nsCOMPtr< nsIMsgFolder > siblingFolder;
> >+		nsCOMPtr< nsIMsgDatabase > msgDatabase;
> >+
> >+		_nntpServer->FindGroup( _groupName.get(), getter_AddRefs(
siblingNewsFolder ) );
> >+		if ( siblingNewsFolder )
> >+			siblingNewsFolder->QueryInterface( NS_GET_IID( nsIMsgFolder ),
getter_AddRefs( siblingFolder ) );
> nsIMsgNewsFolder doesn't inherit from nsIMsgFolder? I hope there's a good
> reason for that :-/ Anyway, use siblingFolder =
> do_QueryInterface(siblingNewsFolder);
> 
> >+				msgDatabase->QueryInterface( NS_GET_IID( nsINewsDatabase ),
getter_AddRefs( newsDatabase ) );
> Again, newsDatabase = do_QueryInterface(msgDatabase);
> 
> >+      msgKey = 0;
> >+        msgKey = NS_STATIC_CAST( nsMsgKey, naturalLong );
> nsMsgKey is just another typedef for unsigned long...
> 
> >+				HandleXRefHeader( xrefHeaderValue, MSG_FLAG_READ == ( aNewFlags &
MSG_FLAG_READ ) );
> != 0 should do here, surely?
> 
> Also, all your whitespace is wrong, remember two spaces indent, no tab
> characters, and if ( NS_FAILED( rv ) ) should be if (NS_FAILED(rv)) etc.
> 


Is the patch the complete file, or just the portions that are changed?  If it's
the complete file, I'm planning on downloading and compiling it tonight (just to
see if it works for me).  If, however, it's just the changes, I'd like some
clarification on the diff view.  Are the portions in Green additional, or
changed material, and the portions in blue are they added or changed material?  

Let me know what I need to do, and I'll resubmit the patch if necessary.  Since,
it's been 5 years since the bug was openend, and there are 200 votes for it. 
And, it's been 8 months since David Bienvenu replied about looking into this.

Bear in mind, that I'm not a C++ Coder in any way, shape, or form.  So, if it's
not ready to submit as a "Polished" final product, then I won't be able to fix it.

Patrick.

[e-mail].

Comment 72

[Patrick Dickey]

(In reply to comment #71)

I've started to clean this patch up a little.  However, the last two or three
changes in the diff file are different from the current build.  So, I'm not
entirely sure where to put them, or how.  Specifically the things I can't find
in order to change are
https://bugzilla.mozilla.org/attachment.cgi?id=136045&action=diff#mozilla.bak/mailnews/news/src/nsNewsFolder.cpp_sec5

https://bugzilla.mozilla.org/attachment.cgi?id=136045&action=diff#mozilla.bak/mailnews/news/src/nsNewsFolder.cpp_sec6

https://bugzilla.mozilla.org/attachment.cgi?id=136045&action=diff#mozilla.bak/mailnews/news/src/nsNewsFolder.cpp_sec7

https://bugzilla.mozilla.org/attachment.cgi?id=136045&action=diff#mozilla.bak/mailnews/news/src/nsNewsFolder.cpp_sec8

I'll attach my copy of the nsNewsFolder.cpp file, so someone else can check my
work and clean it up some more.  As I've mentioned in the earlier post, I'm not
a C++ or C# programmer, so I'd like it if someone proofs this.

Thanks,
Patrick.

Comment 73

[Patrick Dickey]

Attachment: A partially patched version of the current nsNewsFolder.cpp file using the proposed patch in this bug.

This is the partially patched version of the nsNewsFolder.cpp that I referred
to in my latest reply.	You'll note that some of the changes made in the patch
are in different locations from where the patch specified in it's diff view. 
So, it would be nice if someone would check the changes to make sure they'll
work.
Thanks.
Patrick.

Comment 74

[Magnus Melin [:mkmelin]]

Patrick: if you are looking for review you should set the "review?" flag, not
the "review+", which means a review was granted. But, if you are only looking
for general comments, you don't need to set any flat at all. 

Comment 75

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 195605 [details]
A partially patched version of the current nsNewsFolder.cpp file using the proposed patch in this bug.

pdickeybeta, only owners and peers of a module are allowed to give r=. See
http://www.mozilla.org/owners.html for a list of people. Don't ask r for
patched files. Instead attach a cvs diff against the current trunk. And please
read the developer guidelines on mozilla.org which descibes all parts you have
to know or ask on IRC.

Comment 76

[Alexander Kluss]

Execpt marking as read, the messages in the other groups should be receive the flag for replied, forwarded and if possible the Label.

Comment 77

[Karsten Düsterloh]

*** Bug 322276 has been marked as a duplicate of this bug. ***

Comment 78

[Karsten Düsterloh]

*** Bug 323963 has been marked as a duplicate of this bug. ***

Comment 79

[Felix Miata]

removing stale whiteboard and disinterested owner

Comment 80

[Vesa Jääskeläinen]

I looked at a patches and current Thunderbird codebase and thinked how it could be implemented in a good way. As far I know Message-ID is unique on server and all cross-posts contain this same ID. If news message storage database would be modified to store messages by server basis and then change groups to only list what ID's it cointains.

Now if one message is marked as read on one group, then message's flags would be changed to be as read in news storage database. Now when messages would be checked from other groups it would fetch information from single message. Perhaps Xref header would then be read to propagete refresh request to only needed groups to update counts.

This way of doing it would make only one copy of message to disk, would work "always" for all message flags. When listing messages from newsgroup, server tells about new message ID's. These could easily be cross referenced to news message storage database and no need to download messages multipled times.

But it would also have some drawbacks, when news message storage database for server grows large, looking up messages could be slow. This could be fastened if some information would be cached to news groups message lists or more intelligent database would be used. If message is destroyed from all groups then it should also be removed from news message storage database.

Another possibility is what is already discussed on this thread, but it needs to update lot's of different flags to messages in different groups.

Comment 81

[Matti Aarnio]

I do think that one fundamental problem we are hitting here is the MSF database subsystem.   It sure is nice, but does not scale to uses where single message processing needs to poke multiple (newsgroup) databases requiring complete rewrite of each database after small modification.

Better would be any standard binary DB - system already uses Berkeley DB 1.85 (as ancient version as that one is..)

Multi-column databases are, after all, just somehow structured data blob (to present the fields) + subkeys (indexes) to refer to those blobs.

In most cases the system needs:
  - Quick way to ADD things to the DB
  - Quick way to LOOKUP things from the DB

In rare cases it needs also to compact the DB, but that can be separately callable item out of TOOLS menu or whatever.

Using newish SleepyCat DB instead of the ancient Berkeley DB would allow a little bit smarter sub-keyed DB processing - oh,  and  SQLITE (www.sqlite.org) is probably even more appealing with its license and features.

Comment 82

[David :Bienvenu]

>poke multiple (newsgroup) databases requiring complete
>rewrite of each database after small modification.

this isn't true  - Mork has incremental commits, involving a change log at the end of the file, so changing a single property just involves seeking to the end and writing out a few bytes.

Comment 83

[Karsten Düsterloh]

*** Bug 355669 has been marked as a duplicate of this bug. ***

Comment 84

[Karsten Düsterloh]

*** Bug 355671 has been marked as a duplicate of this bug. ***

Comment 85

[Tashfeen Bhimdi]

I've started a campaign for this, $25 already on the bounty:
http://www.dropcash.com/campaign/TBhimdi/implement_mozilla_bug_43278_/

I will send the money off to whomever adds this feature, hopefully before the end of this year.  This is my first time setting up something like this, but I really want this feature added.  Please publicize the campaign to others.  Also, if you start working on this bug, post here so that your work isn't overlapped with somebody else's.

Comment 86

[Tashfeen Bhimdi]

I've closed the campaign.  2 weeks and no donations, so figured it was probably not going to happen.  Makes sense, since it's pretty much extremely "unofficial".

I guess the only it's going to happen is the official way:
Ability for anyone to FUND (Sponsor, bounty) specific bugs (e.g., PayPal)
https://bugzilla.mozilla.org/show_bug.cgi?id=124096

Comment 88

[Arkady Belousov]

(In reply to comment #36)
> also, as the comment you quoted says, that code is a fallback - "useful if the
> server doesn't support Xref properly". Virtually all servers do support Xref, so
> we should use Xref to implement this in the first case (I guess one could also
> implement something message-id based as a fallback, but that's not important).

     If Xref is present - Message-ID equal in all instances. Then what bad in dealing only with Message-ID, and not duplicate efforts in dealing with Xref?

Comment 89

[Ed Murphy]

If the messages are indexed by Xref, but not by Message-ID or (newsgroup + Message-ID), then Xref (when present) would be faster to find.  Dunno about writeback.

I understand not wanting to roll this into the main codebase until it's gold, but until then, could someone turn it into an add-on?

Comment 90

[Stewart Gordon]

How does one make what "gold" now then?

Comment 91

[Joshua Cranmer [:jcranmer]]

Taking a brief look at the patch to implement this, I have a slightly different way to implement this bug that is better IMHO.

Since a message that is crossposted into two (or more)newsgroups is really one message, why not represent it as one message between all the newsgroups?

Rough outline of what would happen:

1. At some point TB gets the Xref header (either through XHDR, ARTICLE, or HEAD depending on when it would be best to parse it).
2. TB parses the Xref header and looks at all the other newsgroups on the server.
3. Does it exist in the other newsgroup?
3a. If not, then add the relevant header to the other newsgroup with a pointer to the current header.
3b. If so, merge the attributes on the other header and then make the second header a pointer to the current header (or t'other way; it doesn't matter).
4. Now in the second newsgroup, when the header is loaded, it gets its attributes from the first group, including read/unread status, killed status, etc.
5. When attributes are changed in the second newsgroup, they changes are pushed to the first newsgroup.

Thoughts?

Comment 92

[Frank Schönheit]

(note: the following is based on my knowledge back when I wrote the "proof of concept - which is some years ago by now. I do not know whether it still applies to the current code, but I assume it does.)
The problem is you need to hook in earlier than when the second header is loaded. When the newsserver node is expanded, TB asks the news server for the read/unread numbers of all newsgroups. All it gets at this point is the article numbers, and it will assume all articles with a previously-unknown number to be unread. Only when the user actually selects a newsgroup, the header will be loaded.

This implies that when a newsgroup contains a yet-unknown message, which is cross-posted, and marked as read in another newsgroup, then expanding the newsserver node will report this unread message, and when the newsgroup is selected, TB recognizes the message as already-read, and the unread-count will silently be corrected. Not really a good user experience, in my opinion.

So, I'd say any solution must take into account TB's caching of the unread-count, which in parts is maintained without fetching any message header.

Comment 93

[Karsten Düsterloh]

(In reply to comment #91)
> Since a message that is crossposted into two (or more)newsgroups is really
> one message, why not represent it as one message between all the newsgroups?

That's the point exactly.
But:

> 1. At some point TB gets the Xref header (either through XHDR, ARTICLE, or 
>    HEAD depending on when it would be best to parse it).

While this would be an enhancement to the current situation, it doesn't help much. A message is the same message only when it has the same message id, looking at arbitrary numbers of a single news server doesn't help:
- locally stored messages don't have Xref numbers
- the same message usually has different Xref numbers for different news servers

The only way of really solving the issue is a profile-global message id index - but that may be too "heavy" when done as msf on old systems (roughly 8 MB file size for 100,000 message ids, as a wild guess? does msf support partial reads of "just the interesting data"?).

(In reply to comment #92)
> This implies that when a newsgroup contains a yet-unknown message, which is
> cross-posted, and marked as read in another newsgroup, then expanding the
> newsserver node will report this unread message, and when the newsgroup is
> selected, TB recognizes the message as already-read, and the unread-count will
> silently be corrected. Not really a good user experience, in my opinion.

This is an inherent problem of "checking just article numbers" vs. "processing the entire message", we have the same problem with filter based message marking etc. The issue *cannot* be solved except by downloading actual headers/messages before telling unread numbers, so it's not that relevant here.


> 
> So, I'd say any solution must take into account TB's caching of the
> unread-count, which in parts is maintained without fetching any message header.
> 

Comment 95

[Magnus Melin [:mkmelin]]

Perhaps Andrew's global database work can be of help...? Or if not, this case might be good to take into consideration during design.

Comment 96

[Joshua Cranmer [:jcranmer]]

(In reply to comment #95)
> Perhaps Andrew's global database work can be of help...? Or if not, this case
> might be good to take into consideration during design.

My current thoughts on the matter amount to the following:
The best way to implement this would be to institute per-account databases. In any case, almost all newsservers provide us with the information we need to know to fix this using XOVER (the last portion is the Xref header, which we don't use), which would allow us to merge articles using the Xref header or fallback to Message-ID guessing in that case.

That being said, it is feasible to write a patch to fix this bug before implementing per-account databases. If anyone has excess time on his/her hands and wishes to implement this under the present configurations, I would not reject the patch out of hand and would even be willing to aid, but it's not my priority to fix this before such time, as I believe that per-account dbs would solve the problem nicely with little extra work.

Comment 97

[David Ascher (:davida)]

Retriaging according to new policy for flags.  

https://wiki.mozilla.org/Thunderbird:Release_Driving

(bugs marked wanted- don't indicate we wouldn't accept patches, but that they're not going to be the focus for release drivers)

Comment 99

[Jan]

This would also be useful for mailing lists (although probably more difficult to implement, as the copies of the messages from the other lists often reside in different folders). Maybe an additional MID database for quickly finding copies would be the only solution there?

Comment 100

[Bill Muskett]

I logged in to check this to see if the fact that someone had changed the title (to read "newsgroups" rather than "groups"), meant that it was actually being seriously worked on at last, but it looks like it's not. Disappointed.

Comment 101

[Leonid S. Knyshov]

This bug is still present in Lanikai 3.1b1

Comment 102

[Chris Ilias [:cilias]]

Until this bug is marked as fixed, everyone can assume that it is present in every release. If you find that it works and this bug is not marked as fixed, then that's something to post about. :-)

Comment 103

[David E. Ross]

This is actually a regression from Netscape 4.  

A capability is broken for which there is no reasonable workaround.  Thus, I'm changing the priority to Major.

Comment 104

[Chris Ilias [:cilias]]

(In reply to comment #103)
> This is actually a regression from Netscape 4.  

That's what the "4xp" keyword is for, thanks.
https://bugzilla.mozilla.org/describekeywords.cgi

Comment 105

[Ant]

Is this bug ever going to be fixed? :(

Comment 106

[Jens Müller (:tessarakt)]

Attachment: Proof of concept patch by Frank Schönheit - v1 (nearly debitrotted)

I applied the patch from attachment 136045 [details] [diff] [review], resolved the conflict and made it buildable again (alas, with still some lines commented out in nsMsgNewsFolder::OnKeyChange).

I don't think it will yet do anything useful. In particular, I think the structure of the notifications has changed slightly - instead of OnKeyChange, there now is OnHdrFlagsChanged and onReadChanged. So I guess the new method in nsMsgNewsFolder will also have to implement one of these handlers (or maybe both).

Comment 107

[Jens Müller (:tessarakt)]

Attachment: Proof of concept patch by Frank Schönheit - v3 (debitrotted more, working somehow)

I now changed the observer method to OnHdrFlagsChanged as defined in nsIDBChangeListener. This works more or less, but only under some conditions:
 - The other newsgroup has to be subscribed already for the message to marked as read there.
 - Marking a message as read in one newsgroup might trigger the header download dialog (the one that asks for the number of headers to download) for the other newsgroup.

So before adressing any cosmetical issues about the patch, I will think about the overall strategy some more.

Comment 108

[Jens Müller (:tessarakt)]

To recapitulate some relevant previous comments:

Comment 17 proposed to get the Xref header through the nsIMsgHeaderSink interface.

Comment 18 and comment 19 introduced the idea to use the Newsgroups: header instead of XRef:

Comment 20 first mentions the read set: "with the article number, you can set it as read getting the read set."

In comment 30, Frank provided a proof-of-concept patch and explained his solution idea:
 - nsIMsgHeaderSink a) is unreliable and b) its purpose is to provide formatted headers to the GUI. Therefore it should not be used.
 - the unread count of other newsgroups should be updated immediately
 - not only the unread->read change should propagate to other newsgroups, but also read->unread changes
The patch works like this:
 - The Xref header is saved in the database.
   ("The main drawback of this is that in order to make this work, the msf files for news folders need to be deleted and rebuilt, and of course increase in size ..." -> I was wondering if there has to be separate migration code?)
 - Observer methods on nsMsgNewsFolder are used to trigger the propagation of the read state.
 - The ReadSet is not used. Instead, the header is retrieved from the other newsgroup and its flags are changed. This also immediately updates the GUI.

Comment 31 notes that for this approach to work, the header has to be already downloaded in the other newsgroup.

Comment 32 explains two other strategies:
 1. a) "When a crossposted message header is downloaded, and the crosspost is to other subscribed 'groups, check whether it has been read in those 'groups and update the status accordingly.  If the header hasn't been downloaded in any other 'groups, leave as unread."
    b) "When a crossposted message has been read, and the crosspost is to other subscribed 'groups, mark it read in the other 'groups."
 2. A global database with message IDs and corresponding statuses.

=> It seems to me that Frank's patch only provides for part b) of solution 1. Adding part a) should not be too difficult. Solution 1 can also be easily generalized to synching of other flag changes, not only unread->read.

In comment 47 Frank explained a concern he had with his patch: He needs to update read/unread counts, but there is no way to to this for a specific newsgroup, only for all newsgroups. He proposes "to extend the nsNNTPProtocol so that it
can also retrieve counts for only a single group."
 => If this is a good idea, I think we should generate a new bug for this, as it is a distinct and quite technical problem.

Frank also mentioned the header download dialog I saw in comment 107.

Comment 80 and comment 81 discuss a database which encompasses more than one newsgroup and thus allows to store the flags of a crossposted message for all its newsgroups. Comment 91 goes into the same direction.
 => @jcranmer: I do not really understand the nature of the "pointer to the current header" mentioned there ... By which identifier/address would that pointer identify the other header? And of course, it has to be a list of pointers, crossposts can go to more than two groups.

In comment 92, Frank explains the problems with updating the unread count: The unread count should be updated even before the header is downloaded in the other newsgroup.
 => Could that be achieved by using the "ReadSet"?

Comment 109

[Jens Müller (:tessarakt)]

My solution idea would be as follows:

On retrieving a new header:
===========================
- Is the message crossposted (check via Xref header)?
  If yes:
  - For each other subscribed newsgroup where the message was crossposted:
    - If the header exists there:
      - Copy flags from the other header.
      - Exit loop ("break;")
  - If the header was not found in another newsgroup:
    - do nothing [Note: message flag changes are not handled here yet.]

On flag changes:
================
- Is the message crossposted?
  If yes:
  - For each other subscribed newsgroup where the message was crossposted:
    - If the header exists there (important: make sure header retrieval is not triggered by this check!):
      - Set changed flags on the other header.
    - If the header does not yet exist there, and if the flag change concerns the read flag:
      - Update the read set.
      - Cause an update of the read count of the other newsgroup only (and not all newsgroups), making sure
        that header retrieval is not triggered.


I hope to get feedback from more knowledgeable developers than me.

Comment 110

[Joshua Cranmer [:jcranmer]]

(In reply to Jens Müller (:tessarakt) from comment #108)
> To recapitulate some relevant previous comments:
> 
> Comment 17 proposed to get the Xref header through the nsIMsgHeaderSink
> interface.

We pull the Xref header during XOVER (it's invariably listed in the results); if we're using HEAD, it's a trivial matter to yank it out. We don't stuff it into the database at the moment, but it's easy enough to do so. This happens whenever we get messages, whereas the header sink is only exercised at message display.

> Comment 20 first mentions the read set: "with the article number, you can
> set it as read getting the read set."

What you really need is the message key, so you can associate it with a header in another database.

> In comment 30, Frank provided a proof-of-concept patch and explained his
> solution idea:
>  - nsIMsgHeaderSink a) is unreliable and b) its purpose is to provide
> formatted headers to the GUI. Therefore it should not be used.
>  - the unread count of other newsgroups should be updated immediately
>  - not only the unread->read change should propagate to other newsgroups,
> but also read->unread changes

I would personally argue that all flag changes (replied, watched, ignored, forwarded, etc.) should be propagated. They're the same message, they should act as the same message. I keep hoping that we'll make traction on a per-account database, where this effect is basically had for free...

> The patch works like this:
>  - The Xref header is saved in the database.
>    ("The main drawback of this is that in order to make this work, the msf
> files for news folders need to be deleted and rebuilt, and of course
> increase in size ..." -> I was wondering if there has to be separate
> migration code?)

You can probably get away without migration: old cross-posts won't work, but after a few months, those won't exist anymore.

> Comment 31 notes that for this approach to work, the header has to be
> already downloaded in the other newsgroup.

This is no longer true, I think, since I made the periodic biff download update the database in... uh... I forget the bug :-) (it was several years ago).

> Comment 32 explains two other strategies:
>  1. a) "When a crossposted message header is downloaded, and the crosspost
> is to other subscribed 'groups, check whether it has been read in those
> 'groups and update the status accordingly.  If the header hasn't been
> downloaded in any other 'groups, leave as unread."
>     b) "When a crossposted message has been read, and the crosspost is to
> other subscribed 'groups, mark it read in the other 'groups."
>  2. A global database with message IDs and corresponding statuses.
> 
> => It seems to me that Frank's patch only provides for part b) of solution
> 1. Adding part a) should not be too difficult. Solution 1 can also be easily
> generalized to synching of other flag changes, not only unread->read.

Solution #2 is what I keep hoping we'll move to eventually, which is why I never put any effort into the bug. Solution #1 is probably achievable by tracking notification of flag changes, but it might work better if it was handled internally to nsNewsDatabase instead of crossing through nsMsgNewsFolder.

> In comment 47 Frank explained a concern he had with his patch: He needs to
> update read/unread counts, but there is no way to to this for a specific
> newsgroup, only for all newsgroups. He proposes "to extend the
> nsNNTPProtocol so that it
> can also retrieve counts for only a single group."
>  => If this is a good idea, I think we should generate a new bug for this,
> as it is a distinct and quite technical problem.

I think this need is also obsoleted by the fact that we actually process news headers instead of just estimated counts.

> Comment 80 and comment 81 discuss a database which encompasses more than one
> newsgroup and thus allows to store the flags of a crossposted message for
> all its newsgroups. Comment 91 goes into the same direction.
>  => @jcranmer: I do not really understand the nature of the "pointer to the
> current header" mentioned there ... By which identifier/address would that
> pointer identify the other header? And of course, it has to be a list of
> pointers, crossposts can go to more than two groups.

Messages are uniquely identified by {folder, message key}. That is the identifier you'd need.

> In comment 92, Frank explains the problems with updating the unread count:
> The unread count should be updated even before the header is downloaded in
> the other newsgroup.
>  => Could that be achieved by using the "ReadSet"?

Again, I think this is obsoleted by the fact that we no longer update the read count without downloading messages in the database.


(In reply to Jens Müller (:tessarakt) from comment #109)
> My solution idea would be as follows:
> 
> On retrieving a new header:
> ===========================
> - Is the message crossposted (check via Xref header)?
>   If yes:
>   - For each other subscribed newsgroup where the message was crossposted:
>     - If the header exists there:
>       - Copy flags from the other header.
>       - Exit loop ("break;")

You still want to make sure filters get applied.

> On flag changes:
> ================
> - Is the message crossposted?
>   If yes:
>   - For each other subscribed newsgroup where the message was crossposted:
>     - If the header exists there (important: make sure header retrieval is
> not triggered by this check!):
>       - Set changed flags on the other header.
>     - If the header does not yet exist there, and if the flag change
> concerns the read flag:
>       - Update the read set.
>       - Cause an update of the read count of the other newsgroup only (and
> not all newsgroups), making sure
>         that header retrieval is not triggered.

This branch is extremely dangerous. If I delete a message from the database, this would incorrectly adjust the read counts. I also feel like edge cases with message expiry would cause issues here. The entire purpose of this "if doesn't exist" branch is preemptive modification of read sets without the database, which (as I've mentioned several times) should no longer be necessary.

Comment 111

[David E. Ross]

There are legitimate reasons to cross-post newsgroup messages.  

One example is the recent message with Subject: SeaMonkey 2.35! cross-posted on 3 September 2015 by Edmund Wong in both mozilla.dev.apps.seamonkey and mozilla.support.seamonkey.  This message announced the release of a new version of SeaMonkey, which is of interest to both users and developers.  

Another example is given by the standard practice of the Big8 Management Board, which cross-posts announcements of proposed changes to newsgroups (add new, delete old, or find a new moderator) in the comp.*, news.*, sci.*, humanities.*, rec.*, soc.*, talk.*, and misc.* hierarchies.  These announcements are generally posted to the affected newsgroup, related newsgroups, news.groups.proposals, and news.announce.newgroups.  The Big8 Management Board operates on the principle of transparency, keeping NNTP users fully informed of changes that might affect them.

Comment 112

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 113

[David E. Ross]

The recent long, cross-posted thread in the news.mozilla.org newsgroups mozilla.support.seamonkey and mozilla.dev.apps.seamonkey regarding the release of SeaMonkey 2.46 clearly illustrates the need for fixing this. 

As I noted in comment #103, this worked in Netscape Communicator 4 and is thus a regression.

Comment 114

[bugzillaroid]

As a workaround, I've recently discovered how to do something
like cross-post management on Thunderbird. It may not meet
every need or desired scenario, but it can at least mark
messages read, or delete them all together, if they've
already been loaded into another folder above it.

1. Set your news folders in the order you want them, with
    the group you want to get a copy of everything on top.
2. For that top folder, no cross-post filter is needed. It
    will get everything.
3. For the next folder below that, from message filters,
    select "customize" from the drop-down list on the far
    left, add "Newsgroups" as a custom field, then select
    that filter from the drop-down list.
4. In the next box, select "Contains"
5. In the next box, put the name of the newsgroup for your  
    top level folder.
6. In "Perform these actions", select "Mark Read", or
    "delete", or whatever. This means that any article which
    was already posted to your top level folder, will now be
    marked read or deleted (or whatever) in the folder
    below it.
7. Repeat these steps for each of the folders below that,
    except add additional rows with the name of the
    newsgroup for each folder that occurs above it, and be
    sure to select "Match any of the following". Now, for
    every folder under the top folder, it will mark read any
    article that has occurred in any of the folders above
    it.

Crude, but it does seem to be working for me. At least you
don't have to see the same articles being unread over and
over again, in every group.

Comment 115

[Daniel]

It's been a long time since I did any programming, but I know this funcitionality was available in Mozilla Suite (or was it Netscape Suite).

In my minds eye, rather that the User having to set their profile up in any particular manner, when I select a News post, TB/SM would exam the Message Header for any line which might indicate that the post has been Cross-posted, e.g. the Xref line or the newsgroups line maybe, who's presences would indicate that the message has been cross-posted and to which news groups on that server that it was cross-posted to.

Then examine that line against the file that contains which news groups (*.rc) on the server that the user is subscribed to, and if any, mark the appropriate message number/s as read.

And, of course, if the message is not cross-posted, mark this message as Read.

Something like ....

If NotExist Xref/newsgroups mark message nnnnn read
else
          Until finished for each group in Xref line
          If exists in ".rc" file NewsgroupName
          Mark Message($num) read
          Next
EndElse
Continue with the rest of the program

E.G. On the news.mozilla.org server, I subscribe to moz.test, moz.gen, moz.support.sm and moz.d.a.sm.
If I download a message from m.test and it has a Xref line which lists m.test(message number), m.s.sm (message number) and m.d.a.sm(message number), those three entries in the news.mozilla.org.rc are set to read
If I download a message from m.test, and that's the only group I subscribe too on n.m.o, its the only message that is marked as read.

Or something like that!! It's been a long, looonng time!!

Comment 116

[Tilman Schmidt]

Somehow I've lost interest in this after 20 years. I'll try to figure out how to stop being notified about the ongoing trickle of CC additions and removals, which seems to be the only thing happening here.