Closed Bug 403970 Opened 14 years ago Closed 9 years ago

Set up an in-house NNTP server and mirror the entirety of news.mozilla.org into it

Categories

(Infrastructure & Operations :: Infrastructure: Other, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: justdave, Assigned: justdave)

Details

We've had a variety of issues with Google Groups that kind of make them untrustworthy as an official source of archives for the mozilla.* hierarchy, from lack of spam removal to messages not showing up periodically, and problems trying to import them when they're noticed missing later.

The only "accurate" source of archives we have at the moment is news.mozilla.org, and that's only accessible via NNTP, which limits the accessibility for a lot of people.  news.mozilla.org is currently hosted on Giganews, which has a stated article retention policy of 4.4 years, which means what we have there now will eventually disappear as well.

The obvious solution for all of this is to set up our own in-house news server, for data retention purposes if nothing else, even if we don't make it accessible.

Sticking a web UI in front of it would be awesome, too, if we could come up with something.
(In reply to comment #0)
> We've had a variety of issues with Google Groups that kind of make them
> untrustworthy as an official source of archives for the mozilla.* hierarchy,
> from lack of spam removal to messages not showing up periodically, and 
> problems trying to import them when they're noticed missing later.

I've experienced all these issues, and if our own news server would enable us to distribute spam canceling privileges more broadly to the base of trusted Mozillians, then that would be a great improvement over the current situation.


> The only "accurate" source of archives we have at the moment is
> news.mozilla.org, and that's only accessible via NNTP, which limits the
> accessibility for a lot of people.  news.mozilla.org is currently hosted on
> Giganews, which has a stated article retention policy of 4.4 years

Hrm, that's another good reason to have our own.


> Sticking a web UI in front of it would be awesome, too, if we could come up
> with something.

Sounds good to me, we should just make sure we continue to play well with Google Groups, even if we set up our own web UI, since many folks will continue to access the newsgroups that way, and it will enable folks to search across both mozilla.* and other hierarchies.
Agree, this should solve a lot of issues regarding the present necessity of moderation and creation of new groups now under consideration.
Status: NEW → ASSIGNED
Status: ASSIGNED → NEW
Changing QA Contact.
QA Contact: justin → mrz
Just am adding this one comment.

As for the spam control, should we go to our own Server. Maybe we could do. Like Microsoft does on their NNTP Server. When a piece a of Spam comes in They give out to Trusted people (those they know will not abuse it) a special new address to forward Spam items to. Then the people in charge of the system at MS views the item and use information to block or knock it out.
Is this a WONTFIX because we can't find a web UI to front this?  We've talked about this several times but always circle back to lack of a UI and I don't see that changing anytime soon.
I'm going to argue that this bug is summarized incorrectly.  Having a canonical, web-accessible archive to these lists is extremely important.  NNTP, on the other hand, seems like an implementation detail.  

The easiest solution here would be to use mailman's built in web archiving functionality, no?  

With a little more work, someone could evaluate other available web-archive-for-mailing-list solutions and see if there is better user-experience to be had that way.

Finally, it's fairly clear that NNTP is a protocol of the past internet and not of the future Internet.  If it makes sense to declare the email / Google Groups versions as canonical at the cost of declaring the NNTP versions "Tier 2" (i.e. not supported as authoritative); we should consider that.
While NNTP may or may not be a thing of the past. It seems to be the most reliable. 
I read and post to various newsgroups and Forums. I find the one with the least problems are NNTP Groups. They hard ever go down unless the server goes up in smoke.

If you want to see a total disaster  try going to adobe's AdobeForums It run by Clearspace Jive. The are problems after Problems. among the most grievous are being signed out in middle or writing a Post, another using email notices, they do not follow sendmail spec and and massges are not threaded properly. Apple's on Support Forums also use a version of Jive and they have the same issues with threading as well. Now a Format that works well is WebX. But it even it has issues. All web based UI have problem of the over head for drawing all the text boxes and UI decoration, takes time and slows down loading of the messages. with NNTP what you see is what you get and it loads fast. loading a GUI style format is painfully slow on slow DSL and Dial-up. Despite what you might think everyone in the world do not have access to DSl,Cable, T1, T%, or FOIS connections. 90 % of the world are still dial up only.
(In reply to comment #6)
> I'm going to argue that this bug is summarized incorrectly.  Having a
> canonical, web-accessible archive to these lists is extremely important.  NNTP,
> on the other hand, seems like an implementation detail.

You seem to feel that newsgroups should be primarily list-servers.
That's not my understanding. In fact some of the problems caused by combining lists and NNTP were caused by forcing that dual functionality.  
> 
> The easiest solution here would be to use mailman's built in web archiving
> functionality, no?  

Why would we need web access if all the posts were saved on the server and not expired too quickly.
At one point in time, you used to be able to access a news url in Firefox, and actually have it handled in NNTP protocol.
 
> With a little more work, someone could evaluate other available
> web-archive-for-mailing-list solutions and see if there is better
> user-experience to be had that way.
> 
> Finally, it's fairly clear that NNTP is a protocol of the past internet and not
> of the future Internet.  If it makes sense to declare the email / Google Groups
> versions as canonical at the cost of declaring the NNTP versions "Tier 2" (i.e.
> not supported as authoritative); we should consider that.

What's with the references to mailing-lists, that's not at all what this bug is about. If a mailing list recipient wants to save his received mail, he is certainly able to do so.

NNTP can be as web-like as you permit it to be, if you take away the multi-functionality of mailing-lists and the restrictions they impose. (plaintext only, don't send that stuff to my inbox)

I never could quite understand why we insist on using Web tools for mail-news development.(Wiki's and on-line forums just tend to spread out the info where someone is sure to miss something.)
(In reply to comment #6)
> I'm going to argue that this bug is summarized incorrectly.  Having a
> canonical, web-accessible archive to these lists is extremely important.  NNTP,
> on the other hand, seems like an implementation detail.  

As I understand it, this bug is not about removing web access (or any other access method), but moving from outsourced solutions to in-house, because the lack of control is causing problems.

> The easiest solution here would be to use mailman's built in web archiving
> functionality, no?  

I assume that doesn't allow posting via the web-interface. Originally, the Google Groups UI for the mozilla.* newsgroups was read-only, but that was changed to allow people to reply to messages when they are given a web address of a message in the archive. See bug 327251.
dmose: it's been Mozilla policy for quite a long time now that all three access methods to the discussion forums (web, email, news) are of equal value and should all work as well as each other, for both reading and writing. Given that each access method continues to have its keen adherents, I don't see that changing any time soon.

I agree that comment #0 is confusing, because its proposed solution doesn't match the problem. If we have a problem with Google Groups, we need to set up an alternative read/write web UI. So in that sense, I agree the bug is mis-summarized. The mailman archive function, as well as generally sucking, is only a read interface and so doesn't fit the bill. 

As I think mrz has said (but seemingly elsewhere), this bug goes round in circles because no-one has yet found alternative web UI software which we can just install to get Google Groups-like function. Until that happens, we are probably going to end up sticking with the solution we have, imperfect as it is.

Gerv
This bug is exactly what the summary says it is, and most of the recent conversation on this bug is off-topic.  We need to get an internal NNTP server set up and mirroring the content, regardless of whether we actually use it or not.  The bug about the issues with Google Groups is bug 425122.  We probably need a new bug for setting up our own web interface.  I think that part's only been discussed in blogs so far.
Assignee: nobody → server-ops
Component: Server Operations: Projects → Server Operations
Whiteboard: [2011q1][tracking]
Assignee: server-ops → justdave
Yes and I would have myself.
Assignee: justdave → server-ops
Whiteboard: [2011q1][tracking] → [2011q2][tracking]
Dave, what needs to happen here? Should we get a VM setup with some NFS storage for this?
Assignee: server-ops → justdave
mrz says no on nntp.
Status: NEW → RESOLVED
Closed: 9 years ago
Component: Server Operations → Server Operations: Infrastructure
QA Contact: mrz → jdow
Resolution: --- → WONTFIX
Whiteboard: [2011q2][tracking]
Corey: is that part of a wider set of decisions on improving/updating our discussion forum framework? If so, is there a documented plan somewhere?

Thanks,

Gerv
Indeed, better discussion facilities are a real pain point (for one example, see bug 716007).  Knowing the what bigger picture direction is here would be tremendously helpful.
(In reply to Gervase Markham [:gerv] from comment #15)
> Corey: is that part of a wider set of decisions on improving/updating our
> discussion forum framework? If so, is there a documented plan somewhere?

Not really..  Trying to be realistic here - this has not become a priority for us enough to make it happen in the last 5 years, and I don't see it rising to the top amidst all of the other stuff coming our way that take priority ahead of all.

If someone handed me a puppet module that would make a rhel6 server become a news.m.o mirror in a single command I'd do it myself.  But getting there isn't on the radar right now.  :(
Component: Server Operations: Infrastructure → Infrastructure: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.