news.mozilla.org not getting all messages

RESOLVED DUPLICATE of bug 327300

Status

mozilla.org Graveyard
Server Operations
P1
critical
RESOLVED DUPLICATE of bug 327300
11 years ago
2 years ago

People

(Reporter: dria, Assigned: justdave)

Tracking

Details

Noticed this morning that a response to a thread didn't seem to make it to Google Groups for archiving, which is a bit annoying because I'm trying to reference that thread in a blog post.

There are three messages in the thread.  The first and third made it to Google Groups, mailing list, and the news groups just fine.

The second message only made it to the mailing list, and didn't make it to Google Groups or the newsgroup.

Link to incomplete Google groups thread:
http://groups.google.com/group/mozilla.dev.mdc/browse_frm/thread/5c27038fddc34dfe/608581ca2a2dfc60#608581ca2a2dfc60

Headers from the missing message:

From: "Justin Wood (Callek)" <callek@gmail.com>
Date: March 8, 2006 12:56:26 AM EST (CA)
To: "Deb Richardson" <deb@dria.org>
Cc: dev-mdc@lists.mozilla.org
Subject: Re: Topic/Category changes summary
Return-Path: <callek@gmail.com>
X-Original-To: deb@dria.org
Delivered-To: deb@mail.off.net
Received: from xproxy.gmail.com (xproxy.gmail.com [66.249.82.207]) by  
mail.off.net (Postfix) with ESMTP id 5B78A77049F for <deb@dria.org>;  
Wed,  8 Mar 2006 00:56:28 -0500 (EST)
Received: by xproxy.gmail.com with SMTP id s10so73304wxc for  
<deb@dria.org>; Tue, 07 Mar 2006 21:56:26 -0800 (PST)
Received: by 10.70.39.7 with SMTP id m7mr412052wxm; Tue, 07 Mar 2006  
21:56:26 -0800 (PST)
Received: by 10.70.6.8 with HTTP; Tue, 7 Mar 2006 21:56:26 -0800 (PST)
Domainkey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com;  
h=received:message-id:date:from:to:subject:cc:in-reply-to:mime- 
version:content-type:content-transfer-encoding:content- 
disposition:references; b=Cixpfx9yfPMhG8f6nX3/HVKLdJ/ 
fyKU9S2gt8WJFKG6Lw8v+nm/Y5FBaRODnZbPtmSxq9tbPYF1Ni7/ 
bVUtqLcwUgJgInM3hyTQpY+pnMC+rYg0falF9QjXhoudFyfqe2 
+hXGGYssdAlEUXVNR6VBDiJA6W9GnNtAqO8VWSWp/4=
Message-Id:  
<2e4780750603072156l27c8e7f6q7ae1f212097f92c3@mail.gmail.com>
In-Reply-To: <7B706523-51C2-4C8A-BF59-08ACEA01029F@dria.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64
Content-Disposition: inline
References: <7B706523-51C2-4C8A-BF59-08ACEA01029F@dria.org>
Google Groups gets its feed from Giganews, so if it didn't make it to news.mozilla.org, it's not going to make it to Google.  Can you forward me the entire message out-of-band?
Summary: Google groups archives are incomplete or not getting all messages → news.mozilla.org not getting all messages
(Reporter)

Comment 2

11 years ago
(In reply to comment #1)
> Google Groups gets its feed from Giganews, so if it didn't make it to
> news.mozilla.org, it's not going to make it to Google.  Can you forward me the
> entire message out-of-band?
> 

Forwarded.

Comment 3

11 years ago
I noticed the message was sent to Deb's address. Is it possible the message didn't make it to the mailing list?  Did anyone else on the list receive the message (via lists.mozilla.org)?
(Reporter)

Comment 4

11 years ago
(In reply to comment #3)
> I noticed the message was sent to Deb's address. Is it possible the message
> didn't make it to the mailing list?  Did anyone else on the list receive the
> message (via lists.mozilla.org)?
> 

the CC is to the list: "Cc: dev-mdc@lists.mozilla.org"

Comment 5

11 years ago
(In reply to comment #4)
> the CC is to the list: "Cc: dev-mdc@lists.mozilla.org"

list.mozilla.org may have filtered it, thus never passing it on to Giganews, thus not making it to Google.

I don't don't why list.m.o would filter it (maybe it was a large message), but finding out if it made it to the list would help in finding the cause.
(In reply to comment #5)
> I don't don't why list.m.o would filter it (maybe it was a large message), but
> finding out if it made it to the list would help in finding the cause.

Yes, it made it to the list. I received this message with the following headers:

From:   callek@gmail.com
Subject: Re: Topic/Category changes summary
Date: March 8, 2006 12:56:26 AM EST (CA)
To:   deb@dria.org
Cc:   dev-mdc@lists.mozilla.org
X-Eon-Dm: dm30
Return-Path: <dev-mdc-bounces@lists.mozilla.org>
Received: from lists.mozilla.org (207.126.111.250 [207.126.111.250]) by dm04.mta.everyone.net (EON-INBOUND) with ESMTP id dm04.44040ac6.5148b63 for <beltzner@mozilla.com>; Tue, 7 Mar 2006 21:56:29 -0800 (PST)
Received: from notorious.mozilla.org (localhost.localdomain [127.0.0.1]) by lists.mozilla.org (Postfix) with ESMTP id EAF0F5800E; Tue,  7 Mar 2006 21:56:28 -0800 (PST)
Received: from xproxy.gmail.com (xproxy.gmail.com [66.249.82.205]) by lists.mozilla.org (Postfix) with ESMTP id AFDE058004 for <dev-mdc@lists.mozilla.org>; Tue,  7 Mar 2006 21:56:26 -0800 (PST)
Received: by xproxy.gmail.com with SMTP id s12so69365wxc for <dev-mdc@lists.mozilla.org>; Tue, 07 Mar 2006 21:56:26 -0800 (PST)
Received: by 10.70.39.7 with SMTP id m7mr412052wxm; Tue, 07 Mar 2006 21:56:26 -0800 (PST)
Received: by 10.70.6.8 with HTTP; Tue, 7 Mar 2006 21:56:26 -0800 (PST)
X-Original-To: dev-mdc@lists.mozilla.org
Delivered-To: dev-mdc@lists.mozilla.org
Domainkey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Cixpfx9yfPMhG8f6nX3/HVKLdJ/fyKU9S2gt8WJFKG6Lw8v+nm/Y5FBaRODnZbPtmSxq9tbPYF1Ni7/bVUtqLcwUgJgInM3hyTQpY+pnMC+rYg0falF9QjXhoudFyfqe2+hXGGYssdAlEUXVNR6VBDiJA6W9GnNtAqO8VWSWp/4=
Message-Id: <2e4780750603072156l27c8e7f6q7ae1f212097f92c3@mail.gmail.com>
In-Reply-To: <7B706523-51C2-4C8A-BF59-08ACEA01029F@dria.org>
Mime-Version: 1.0
References: <7B706523-51C2-4C8A-BF59-08ACEA01029F@dria.org>
X-Beenthere: dev-mdc@lists.mozilla.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: "Discussion list for all things related to the Mozilla Developer Center, including developer documentation." <dev-mdc.lists.mozilla.org>
List-Unsubscribe: <https://lists.mozilla.org/listinfo/dev-mdc>, <mailto:dev-mdc-request@lists.mozilla.org?subject=unsubscribe>
List-Post: <mailto:dev-mdc@lists.mozilla.org>
List-Help: <mailto:dev-mdc-request@lists.mozilla.org?subject=help>
List-Subscribe: <https://lists.mozilla.org/listinfo/dev-mdc>, <mailto:dev-mdc-request@lists.mozilla.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1766732003=="
Sender: dev-mdc-bounces@lists.mozilla.org
Errors-To: dev-mdc-bounces@lists.mozilla.org
(In reply to comment #2)
> (In reply to comment #1)
> > Google Groups gets its feed from Giganews, so if it didn't make it to
> > news.mozilla.org, it's not going to make it to Google.  Can you forward me the
> > entire message out-of-band?
> > 
> 
> Forwarded.

I never got this, could you try again? (have you seen any more?)

I suspect Giganews' spamfilters probably nailed it.
(Reporter)

Comment 8

11 years ago
Forwarded again.
Have you seen any more that haven't shown up?  I don't see any information in the mail you forwarded that looks like something giganews would block...  hopefully it was a fluke...
Assignee: server-ops → justdave
I'm going to close this out for lack of being able to reproduce.  If you run across any others, feel free to reopen it.
Status: NEW → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → WORKSFORME
Yup, I got some more examples: I started a thread (via email) yesterday in mozilla.dev.planning. Later, I replied to a reply of that thread. The original message was delivered to Google Groups and by mailman, but did not show up in NNTP. The later reply was only delivered in mailman, and doesn't appear in NNTP or Google Groups:

ORIGINAL MESSAGE -- EMAIL HEADERS:

 X-Eon-Dm: dm10
Return-Path: <dev-planning-bounces@lists.mozilla.org>
Received: from lists.mozilla.org (207.126.111.250 [207.126.111.250])
	by dm10.mta.everyone.net (EON-INBOUND) with ESMTP
	id dm10.4419136a.1e159c3; Thu, 23 Mar 2006 15:17:09 -0800 (PST)
Received: from notorious.mozilla.org (localhost.localdomain [127.0.0.1])
	by lists.mozilla.org (Postfix) with ESMTP id 1F5B05801C;
	Thu, 23 Mar 2006 15:17:09 -0800 (PST)
X-Original-To: dev-planning@lists.mozilla.org
Delivered-To: dev-planning@lists.mozilla.org
Received: from smtp02.bis.na.blackberry.com (smtp02.bis.na.blackberry.com
	[216.9.248.49])
	by lists.mozilla.org (Postfix) with ESMTP id 59AAF1336D
	for <dev-planning@lists.mozilla.org>;
	Thu, 23 Mar 2006 15:17:07 -0800 (PST)
Message-ID: <1092075651-1143155826-cardhu_blackberry.rim.net-3248-@engine11-cell02>
Sensitivity: Normal
Importance: Normal
To: "bonecho" <bonecho@mozilla.org>,
	"dev.planning" <dev-planning@lists.mozilla.org>
From: "Mike Beltzner" <beltzner@mozilla.com>
Date: Thu, 23 Mar 2006 23:15:55 +0000 GMT
MIME-Version: 1.0
Cc: 
Subject: Fx2 Planning Centre (was PRD review)
X-BeenThere: dev-planning@lists.mozilla.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: dev-planning.lists.mozilla.org
List-Unsubscribe: <https://lists.mozilla.org/listinfo/dev-planning>,
	<mailto:dev-planning-request@lists.mozilla.org?subject=unsubscribe>
List-Post: <mailto:dev-planning@lists.mozilla.org>
List-Help: <mailto:dev-planning-request@lists.mozilla.org?subject=help>
List-Subscribe: <https://lists.mozilla.org/listinfo/dev-planning>,
	<mailto:dev-planning-request@lists.mozilla.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============1585748431=="
Sender: dev-planning-bounces@lists.mozilla.org
Errors-To: dev-planning-bounces@lists.mozilla.org

ORIGINAL MESSAGE - GOOGLE GROUPS URI: http://groups.google.com/group/mozilla.dev.planning/browse_frm/thread/363d9531af2dad10/69a04ae0b1032661#69a04ae0b1032661

ORIGINAL MESSAGE - NNTP: missing in action!

FOLLOW-ON MESSAGE - EMAIL HEADERS:

X-Eon-Dm: dm01
Return-Path: <dev-planning-bounces@lists.mozilla.org>
Received: from lists.mozilla.org (207.126.111.250 [207.126.111.250])
	by dm01.mta.everyone.net (EON-INBOUND) with ESMTP
	id dm01.441b0d44.30505a8; Thu, 23 Mar 2006 17:05:38 -0800 (PST)
Received: from notorious.mozilla.org (localhost.localdomain [127.0.0.1])
	by lists.mozilla.org (Postfix) with ESMTP id 1585F5808D;
	Thu, 23 Mar 2006 17:05:38 -0800 (PST)
X-Original-To: dev-planning@lists.mozilla.org
Delivered-To: dev-planning@lists.mozilla.org
Received: from smtp03.bis.na.blackberry.com (smtp03.bis.na.blackberry.com
	[216.9.248.50])
	by lists.mozilla.org (Postfix) with ESMTP id 2DFFF13E85
	for <dev-planning@lists.mozilla.org>;
	Thu, 23 Mar 2006 17:05:35 -0800 (PST)
Message-ID: <195139708-1143162334-cardhu_blackberry.rim.net-10754-@engine27-cell02>
References: <1092075651-1143155826-cardhu_blackberry.rim.net-3248-@engine11-cell02>
	<f4e041d00603231639t4d7a420esca70b5fbb8fe1bf8@mail.google.com>
In-Reply-To: <f4e041d00603231639t4d7a420esca70b5fbb8fe1bf8@mail.google.com>
Sensitivity: Normal
Importance: Normal
From: "Mike Beltzner" <beltzner@mozilla.com>
Date: Fri, 24 Mar 2006 01:05:28 +0000 GMT
MIME-Version: 1.0
Cc: "dev.planning" <dev-planning@lists.mozilla.org>,
	bonecho <bonecho@mozilla.org>
Subject: Re: Fx2 Planning Centre (was PRD review)
X-BeenThere: dev-planning@lists.mozilla.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: dev-planning.lists.mozilla.org
List-Unsubscribe: <https://lists.mozilla.org/listinfo/dev-planning>,
	<mailto:dev-planning-request@lists.mozilla.org?subject=unsubscribe>
List-Post: <mailto:dev-planning@lists.mozilla.org>
List-Help: <mailto:dev-planning-request@lists.mozilla.org?subject=help>
List-Subscribe: <https://lists.mozilla.org/listinfo/dev-planning>,
	<mailto:dev-planning-request@lists.mozilla.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============0353668301=="
Sender: dev-planning-bounces@lists.mozilla.org
Errors-To: dev-planning-bounces@lists.mozilla.org
To: undisclosed-recipients: ;

FOLLOW-ON MESSAGE - GOOGLE GROUPS URI: missing in action!

FOLLOW-ON MESSAGE - NNTP: missing in action!

some notes: the follow on message was "To: undisclosed recipients;" (which is another wierd thing that happens a lot) and CC'd to dev-planning@lists.mozilla.org ... could that be part of the (but obviously not the entire) problem?
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
cc'ing a bunch of people; justin to make sure he's getting the email on this, dbaron, ben, cbeard and mitchell because it's very much related to ongoing decisions about moving discussion to forums that use this infrastructure. Obviously we need to ensure that if we SAY that people can get to these discussions through email, NNTP or Google Groups, then we need to make sure that they are getting those discussions reliably.
This may or not be related - I just received several messages on the mailing list from yesterday... may be related or it may just be latency
That can also be because of someone going through the mailman moderation queue -- I just cleared the moderation queue on dev-planning, if that's the list you're talking about.
(In reply to comment #12)
> cc'ing a bunch of people; justin to make sure he's getting the email on this

he's already the qa contact ;)

Beltzner's additional info was enough to prove there's something real going on here, I've opened a ticket with Giganews.
Whiteboard: waiting response from Giganews
response from Giganews:

---8<---
That article was filtered from the text group as a binary article, since the article was base64 encoded.  We'll still feed that stuff on to Google (we feed everything to Google and let them filter what they want), but that will be seen as a binary and filtered.

If you could, recommend to people that they not base64-encode their text articles.  Post them as unencoded text when posting to text groups.  We don't check the actual content of the binary to see if it's really text or not (at the volume we feed our servers, that'd be crushing), and we don't trust content-type headers, since those can easily be forged.  Basically, if it looks like binary, regardless of how "binary" it truly is, it will be dropped in a text group.
---8<---

That makes this a duplicate of bug 327300, but I'll leave this open for a day or so for additional comment first.
Whiteboard: waiting response from Giganews
There's no attachment here, so it's not a dup of that bug, and some mail clients (like that on the blackberry) don't give a choice about how the body of the message is encoded.  (I suspect it uses base64 so that it doesn't have to worry about gateways that don't like high-bit characters, since it's a perfectly legal and reasonable way to send mail supported by pretty much every mail client the recipient might have.)

Can we get giganews to turn off this "protection" for us on our groups, or to honour MIME types, or to send it to the moderator address (thus permitting posts from people who are subbed to the list, at least, though that is only marginally less ridiculously painful) or _something_?
Or, heck, _we_ could decode base64 for messages that have text MIME types before we send it to the news gateway, which wouldn't require us to get Giganews to do anything at all.

(That moderator-address thing won't work, I now realize; I apologize for the dumbness there.)
(In reply to comment #18)
> Or, heck, _we_ could decode base64 for messages that have text MIME types
> before we send it to the news gateway, which wouldn't require us to get
> Giganews to do anything at all.

If you actually read the other bug, that's part of it ;)

I did read the other bug, and it sounds like it should be WONTFIXED, or summarizes another problem entirely.
Priority: -- → P1
OK, I asked Giganews, and they will not budge on this.  Bug 327300 is the only solution as long as we're still using them.

*** This bug has been marked as a duplicate of 327300 ***
Status: REOPENED → RESOLVED
Last Resolved: 11 years ago11 years ago
Resolution: --- → DUPLICATE
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.