Newsgroups are not being propagated to Google Groups

RESOLVED FIXED

Status

mozilla.org
Discussion Forums
P1
normal
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: ianbicking, Assigned: mburns)

Tracking

Details

(Whiteboard: [waiting for Google Groups])

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
Recent posts to mozilla.dev.platform, mozilla.dev.webapps, and mozilla.dev.b2g are not being propagated to their respective Google Group mirrors (maybe all groups, those are just the ones we checked, all of which have had missing posts).  The confirmed missing post is by fabrice@mozilla.com though I believe there are other missing posts.
Someone will need to contact Google.

Gerv
Assignee: gerv → server-ops
(Assignee)

Updated

6 years ago
Blocks: 711473
(Assignee)

Updated

6 years ago
Blocks: 708511
(Assignee)

Updated

6 years ago
Blocks: 701127
(Assignee)

Updated

6 years ago
Assignee: server-ops → mburns
(Assignee)

Comment 2

6 years ago
Sent a second email to Google Groups admins noting sync issues. Will wait a couple business days before further escalation up the ladder.
Status: NEW → ASSIGNED
(Reporter)

Comment 3

6 years ago
I'm also subscribed here, and haven't been gotten any messages: https://lists.mozilla.org/listinfo/dev-webapps - unfortunately I don't know if I should have gotten any messages (well, I at least have written to that email address, but I'm not sure I get my own messages, and there's no archive except the Google Group).
(Assignee)

Updated

6 years ago
Blocks: 717589
(Reporter)

Comment 4

6 years ago
After checking via a newsreader, mail sent to the dev-webapps list goes to the group, but traffic on the webapps group does not come back to me on the dev-webapps list.
(Assignee)

Updated

6 years ago
No longer blocks: 717589
(Assignee)

Comment 5

6 years ago
mozilla.dev.marketplace has also reported this problem.

Comment 6

6 years ago
mozilla.dev.general.zh has faced a similar problem that google group page stayed empty weather we posted via the mailing-list, news-server or directly post on google group page, while the mailing-list and news-server works normally.

Comment 7

6 years ago
Has this been escalated, per comment #2 above? It is a bit ridiculous how long this is taking. Is this what every new group we create has to go through if we want it gatewayed to google groups?

Comment 8

6 years ago
(In reply to Ragavan S [:rags] from comment #7)
+1, we've been waiting for 4+ month, it's ridiculous that Google always not reply.
mburns: ping?

Gerv

Comment 10

6 years ago
mburns: any update?
(Reporter)

Comment 11

6 years ago
The real problem is that we don't have any public archives of the group, the only way to read old messages is to connect with a newsreader.  Can we setup pipermail or some other simple mail archiver so at least we can have something visible?
Can't we do a news-mailman gateway if the google groups gateway's broken?

Upping prio, as this is going to become problematic for the apps group with the upcoming marketplace launch.
Priority: -- → P2
(Reporter)

Comment 13

6 years ago
We do have a working mail gateway (I don't have the link on me) but it is missing archives (it just points to google groups)
Ah, right.  The link is https://lists.mozilla.org/listinfo/dev-webapps.

Comment 15

6 years ago
(In reply to Ian Bicking (:ianb) from comment #11)
> The real problem is that we don't have any public archives of the group, the
> only way to read old messages is to connect with a newsreader.  Can we setup
> pipermail or some other simple mail archiver so at least we can have
> something visible?

Agree, we just need an working archive of the discussion on mailing list, if the Google solution is so lacking of maintaining (for monthes!), find another way is better solution.

Comment 16

6 years ago
Can someone from IT/serverops reply here? The last update I see is from Jan 23rd. Can you at least help with a news-mailman gateway like David suggests in comment #12? I don't care it's on google groups, just get me an archive.

Bumping the priority to P1 to see if we can at least get a response from someone.
Priority: P2 → P1
(Assignee)

Comment 17

6 years ago
We've sent in a number of emails (ever couple weeks), and rbryce called at least once, as I understand it. The source of delay here was that we (I) was too optimistic toward Google's eventual response.

AFAIK, the next escalation step is having Mitchell call up a google contact and have them get the ball rolling from that end.

CC'ing Phong: what needs to happen to get action from Google Groups on this?
(Assignee)

Updated

6 years ago
Blocks: 728054
(Assignee)

Comment 18

6 years ago
I google groups admin responded to my latest email poke:

"We're aware of the problem, and a solution should be in place by the end of the month - we had to rewrite some ancient nntp code to make it compatible with current google infrastructure.
-Andrew Shebanow"
(Assignee)

Comment 19

6 years ago
YOU get a newsgroup that syncs to Google Groups and YOU get a newsgroup that syncs to Google Groups. EVERYBODY GETS NEWSGROUPS THAT SYNCS TO GOOGLE GROUPSSSS! http://bit.ly/gcSoTx



https://groups.google.com/group/mozilla.dev.platform/topics

https://groups.google.com/group/mozilla.dev.webapps/topics

https://groups.google.com/group/mozilla.dev.b2g/topics

https://groups.google.com/group/mozilla.dev.marketplace/topics

https://groups.google.com/group/mozilla.dev.general.zh/topics

et cetera
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
It looks like some of us get a newsgroup that syncs to Google Groups only messages from March 9 onwards (at least webapps, but I suspect also marketplace and general.zh), while others get one that syncs all messages (platform, b2g).  Might all of us get the latter type of newsgroup?

Webapps in particular has messages dating back at least to January that would be super-awesome to have available for posterity (and because people keep asking us what we said back then).
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
mozilla.community.games appears to not exist on Google Groups at all.  

Perhaps there would be some advantages to driving our use of Google groups via the mailing list instantiations of our fora rather than the newsgroup versions.  Some that come to mind include:

* it would be possible to easily set these up by hand individually rather the depending on third-parties
* it shouldn't, in theory, be too hard to automate that setup ourselves
* given the relative prominence of mailing lists versus newsgroups in the world at large (ie not just among techies), I would bet that Google gives implementation, testing, and maintenance priority to the parts of the Groups code that touches email over the parts that touch NNTP.
(In reply to Dan Mosedale (:dmose) from comment #21)

> Perhaps there would be some advantages to driving our use of Google groups
> via the mailing list instantiations of our fora rather than the newsgroup
> versions.

Some Labs projects have set up Google Groups, which gives them mailing lists and web interfaces but not newsgroups.  It works great.  We should do more of that.

Nevertheless, we should also make sure the existing mailing list/newsgroup groups are properly archived.

Comment 23

6 years ago
(In reply to Myk Melez [:myk] [@mykmelez] from comment #22)
> Some Labs projects have set up Google Groups, which gives them mailing lists
> and web interfaces but not newsgroups.  It works great.  We should do more
> of that.

Actually, it closes out a part of the community, people like me who hate signing up for a ton of lists and get their accounts spammed with stuff they only want to glance over, which NTTP allows beautifully. It also closes out anyone who doesn't want to possess or use Google accounts, which after the recent privacy changes at Google is a growing audience.
I framed my suggestion the way I did specifically because I don't think it's productive to get into a "usefulness of NNTP" discussion here; that's very much off-topic for this bug.  

I've specifically suggested changing the technical detail of which channel gets used to populate the Google groups.  I'd be interested to hear from someone in IT about the technical merits of that strategy w.r.t. solving the particular problem we're having here.
(Assignee)

Comment 25

6 years ago
You are quite right.

I've sent in an email to the Google Groups admin. Will let you know if they do/don't respond.
Great; thanks!
(Assignee)

Comment 27

6 years ago
No response as of yet. I just sent in another request for help.
Thanks muchly for staying on top of the Google folks, getting this fixed will be helpful to a bunch of folks.
(Assignee)

Updated

6 years ago
Whiteboard: [waiting for Google Groups]
We've just had someone from the community ask why the games archive link is broken, what's the next step in escalating this?  We really need a more functional discussion system than we've currently got.
Agreed, this has now been down for almost 2 quarters.  I think it's time to start considering a wider range of solutions.
(Reporter)

Comment 31

5 years ago
We also cannot have unarchived mail, it makes it very hard to bring people into conversations, and decisions are essentially unaccountable when there's no artifact of the discussion.  If Google Groups isn't entirely reliable we need a backup, such as pipermail.  There's no reason we can't be archiving to both at the same time.
Google Groups proper is actually entirely reliable, it's the Mozilla-specific newsgroup/mailing list/Google Group triad that isn't reliable.  And Google seems little interested in continuing to maintain it, much less enhance it, based on the lack of response to the issues raised in this bug.

Mozilla projects that use just Google Groups, like Jetpack <https://groups.google.com/forum/#!forum/mozilla-labs-jetpack>, have none of these problems, and we should use Google Groups by itself for new projects (and switch to it for existing ones having problems with the triad).
Do people need a google account to use google groups?
I believe for reading google groups, one does not need an account.  To subscribe or post one probably does.

Comment 35

5 years ago
There's actually two different things that are both called "Google Groups" and hosted on the same site: A Google-hosted mailing list service and a web interface to newsgroups. The former is working pretty well and AFAIK even needs an account to read messages, of course needs one to subscribe and post. The latter, which we are using here, has always been sucky, but has been degrading even more over time, and has free access to reading messages - needs an account to post as well, though.
Unfortunately, nobody seems to ever have built a really good web interface to newsgroups (though I hear the gmane interface is pretty usable for their own newsgroups that are mirroring mailing lists), which is probably one reason why their usage is on a steady decline. Standalone clients for messaging are the only really good access to NNTP/newgroups and standalone client usage for messaging is declining - even if the Thunderbird team doesn't want to hear that, understandably.
(Assignee)

Updated

5 years ago
Depends on: 770423
(Assignee)

Updated

5 years ago
(Assignee)

Updated

5 years ago
Blocks: 772092
(Assignee)

Updated

5 years ago
No longer blocks: 772092
(Assignee)

Updated

5 years ago
Blocks: 772872
Kev, I see that you're already cced. Do you have any Google contacts who can help escalate this growing issue?

Comment 37

5 years ago
I just happened to stumble upon this bug after reading a Google Alert pointing at a blog post about it.

I run part of the service at Google that this depends on, and although we've gotten your emails in the past several months, and I made sure your groups were added, apparently there was a second place they needed to get added to work.

I fixed that yesterday, and I see that the mozilla.community.games group has now been created.

The usenet portion of Groups has been kept together with spit and bailing wire, and we ran out of spit about 9 months ago.  We're still in the middle of moving the whole system over to a new system... but there's still something quite hacky about the way that new groups are transferred as control messages over usenet, and it may take us some time to get everything right in the new system.  We hope things will be stable on the new system in the next 4-6 weeks.

You might have more luck with premium Google Apps for Your Domain, which would give you access to siloed mailing lists with an API for managing them (afaik), and Groups also has an "external archive" mode that may work as well.  We're happy to try and keep the current system working, though.

We've added monitoring for our at least the portion of the system that was broken that I just fixed... but the whole deal still depends on a lot of steps, so please continue to notify us if things aren't working.  We'll try and do a better job of making sure the exact group gets created and messages are flowing.
Thanks for jumping in here Brandon. I see that at least some groups still haven't been created. Can you follow up on the rest of the specific requests in this bug? From the dependencies, I think the complete list is:

mozilla.community.antarctica (not sure of the status of this list on our end)
mozilla.community.morocco
mozilla.community.vietnam
mozilla.compatibility
mozilla.dev.chat
mozilla.dev.sumo
mozilla.support.instantbird
mozilla.webapps
mozilla.webapps.pt-br
mozilla.education has been revived, but only appears as an archived newsgroup on Google Groups.
(In reply to Brandon Long from comment #37)
> I run part of the service at Google that this depends on, and although we've
> gotten your emails in the past several months, and I made sure your groups
> were added, apparently there was a second place they needed to get added to
> work.
> 
> I fixed that yesterday, and I see that the mozilla.community.games group has
> now been created.

In addition to the problem of groups not being created, at least dev-webapps has two other issues:

1. messages from before the group was created never made it to the group;
2. some messages from after the group was created don't make it to the group.

An example of the latter problem is the One-app-per-origin thread <https://groups.google.com/forum/#!topic/mozilla.dev.webapps/lILc00-owaQ>, which contains 30 messages, only 18 ever made it to the group.

That's a big problem, because such conversations lead to decisions that entrain implementations (in this case bug 775847), and we need to be able to refer back to them to know what we're doing and how to do it.


> You might have more luck with premium Google Apps for Your Domain, which
> would give you access to siloed mailing lists with an API for managing them
> (afaik), and Groups also has an "external archive" mode that may work as
> well.  We're happy to try and keep the current system working, though.

We did something like that for the Mozilla Labs groups, and it worked great, as I noted in comment 22.  I think we should do it for new groups, even though some Mozillians don't like that approach (cf. comment 23).

However, we should still make sure our existing groups work properly.


> We've added monitoring for our at least the portion of the system that was
> broken that I just fixed... but the whole deal still depends on a lot of
> steps, so please continue to notify us if things aren't working.  We'll try
> and do a better job of making sure the exact group gets created and messages
> are flowing.

Thanks Brandon, I really appreciate it!  In addition to making sure the new groups get created, can you look into the issues of messages not making it to them?

Comment 41

5 years ago
With usenet, messages sent before the newsgroup is created are rejected, so we don't have them.

The newsgroup creation itself is handled asynchronously, ie in a batch job I think once a day, so any messages sent immediately after the control message are also going to be rejected.

That thread is from April and I have no way of looking into the missing messages from that thread, any logs are long gone.

Do you have a more recent example of messages that didn't propagate?
Created attachment 645805 [details]
dev-webapps message that didn't propagate

(In reply to Brandon Long from comment #41)
> Do you have a more recent example of messages that didn't propagate?

Here's a recent message that didn't propagate.

Comment 43

5 years ago
Still waiting for the group creation!
mozilla.community.morocco
(Assignee)

Updated

5 years ago
Blocks: 761273

Comment 44

5 years ago
If someone can get me a list of newsgroups that need to be added/deleted and their moderation status, I'm happy to update our list.

Also, the new system is mostly in place at the moment, so I can better look for propagation errors, at least for anything in the last 8 days (the length of our logs).
Thanks for helping out Brandon. I listed 9 groups in comment 38 that still need to be created. Can you clarify what you need for moderation status?
(Assignee)

Comment 46

5 years ago
Brandon et al.,

I've sent you a list of all the newsgroups I have with open bugs (those listed in comment 38, plus a couple from other bugs I own), including their moderation status. Hopefully this is an exhaustive list (15, for those playing along at home). Thank you!

Comment 47

5 years ago
I received the list of groups and updated them in our datastore and verified they were all active/available from the Groups web interface.  Things should be working now.
(Assignee)

Comment 48

5 years ago
mozilla.community.morocco
mozilla.community.vietnam
mozilla.compatibility

all have had messages show up correctly and appear to be working fine. A few  more newsgroups have been created on Google Groups, but I don't see messages flowing yet. Still investigating.
(Assignee)

Comment 49

5 years ago
m.c.morocco, m.c.vietnam and m.compatibility have all been working nicely.

We only have a couple outstanding lists (.usa and .switzerland, .comms and .education) that have issues. Waiting to hear back from Google on why those haven't been picked up in their Monday scans.
(Assignee)

Updated

5 years ago
(Assignee)

Updated

5 years ago
Blocks: 804370
(Assignee)

Comment 50

5 years ago
Brandon Long from Google has been in contact with me, and should be adding all outstanding lists to date. Yay!
(Assignee)

Updated

5 years ago
Blocks: 817014

Comment 51

5 years ago
Hi

The mailing list now works, thanks!

I put the language in French and the list info page lost the mozilla look, is not the most important but, can you help me?

Finaly, can any add the mailing list to www.mozilla.org/about/forums/

Best regards
(Assignee)

Updated

5 years ago
Blocks: 791666
(Assignee)

Comment 52

5 years ago
I believe the problem you saw with setting French as the language is bug 815466, which I hope to fix. I'm not sure where mozilla.org/about/forums/ gets populated from. It might be by way of Google Groups, in fact. Let me ask around.
http://www.mozilla.org/about/forums is populated by a script run over a list of newsgroups which I maintain; I have help from other scripts to maintain it, and those scripts consult the NNTP server to see what groups are valid.

I still need to update it manually, though. I'll add it to my ToDo list.

Gerv

Comment 54

5 years ago
Hi, Tanks to all for your help, now google groups work and the template is in the bug 815466, I think that the only issue for close this tiket is the inclusion of Québec community in /about/forums :)
Check about/forums :-)

Gerv
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago5 years ago
Resolution: --- → FIXED
(Assignee)

Updated

5 years ago
Blocks: 824945

Comment 56

5 years ago
Thanks and happy new year!!

Updated

5 years ago
Blocks: 846608

Updated

5 years ago
No longer blocks: 846608
You need to log in before you can comment on or make changes to this bug.