325854 - Not all news.mozilla.org posts are on Google

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Chris Ilias [:cilias]]

This is probably an issue between Giganews and Google. Not all posts on news.mozilla.org are making it to Google. So far, it seems random.
For instance:
The "2006-01-30 - Summary of mozilla.org staff meeting" post by Gerv in mozilla.dev.planning did not make it to google, but "2006-01-16 - Summary of mozilla.org staff meeting", which was posted just two minutes earlier did make it. And a response to the post that did not make it, /was/ archived on Google.

Comment 1

[Dave Miller [:justdave]]

I've forwarded this to both Giganews and Google.

Comment 2

[Dave Miller [:justdave]]

response from Giganews:
============================
Looking through the group on our servers, I found the corresponding message ID for both articles in question:

"2006-01-30 - Summary of mozilla.org staff meeting"
<[e-mail]>

"2006-01-16 - Summary of mozilla.org staff meeting"
<[e-mail]>

I checked against Google's server that we feed to, it returns it has already seen those two message IDs, which means we've already passed them on to them.

200 news4.google.com InterNetNews server INN 2.4.2 (20041025 prerelease)
ready
IHAVE <[e-mail]>
435 Duplicate
IHAVE <[e-mail]>
435 Duplicate
quit
205 .

When I go to Google Groups and browse mozilla.dev.planning, I only see two articles: from Nitin at Feb 4, 1:05 am, and Philip at Feb 3, 12:58 pm.

When I use the 'Advanced Search' on Google Groups to do a message ID lookup, it can not find either message:

The requested message, <[e-mail]>, could not be found.
The requested message, <[e-mail]>, could not be found.

This tells me that we've correctly passed on the articles to Google, and for whatever reason Google is not finding/displaying them.

Comment 3

[Dave Miller [:justdave]]

Anyone have any recent examples of this?  Google thinks the existing examples are too old to prove anything is amiss...

Comment 4

[Chris Ilias [:cilias]]

I did some tests in mozilla.test on Monday.
Message:<[e-mail]> did not show on Google.

I'll see if I can catch anything more recent, as well as do another series of tests.

Comment 5

[Frank Wein [:mcsmurf]]

Some post in mozilla.dev.builds is missing:
Date: Mon, 06 Feb 2006 08:11:36 +0800
Message-ID: <[e-mail]>

and possibly in mozilla.dev.tech.layout some postings are also missing (but maybe they still show up in Google Groups since they are quite new):
Message-ID: <[e-mail]>
Message-ID: <[e-mail]>

Comment 6

[Chris Ilias [:cilias]]

In mozilla.dev.mozilla.org <[e-mail]> was posted 4 hours ago. It's not on Google.

Comment 7

[Dave Miller [:justdave]]

Thanks, I've sent these along.

Comment 8

[Dave Miller [:justdave]]

Response from Google:

---8<---
I looked at it today, but have not found the reason yet.

A couple of our edge servers _sometimes_ were not propagating
articles further after getting them from Giganews, so the
articles were not indexed. I have not seen anything like
that before, and I confirmed it happens only for some
mozilla.* articles (all other hierarchies are ok). Maybe INN
is "getting confused" because of the special mozilla.*
configuration.

I restarted the servers and have not found any missing article
since from 9:51am PST today.
---8<---

Comment 9

[Chris Ilias [:cilias]]

I guess restarting the servers hasn't completely fixed it.
Message id: <[e-mail]> in mozilla.support.firefox is missing.

Comment 10

[Dave Miller [:justdave]]

Feb 12 10:37:31.028 - news-out.nntp.giganews.com <[e-mail]> 437 Article posted in the future -- "Mon, 13 Feb 2006 11:42:21 -0700"

Comment 11

[Chris Ilias [:cilias]]

I've been trying pretty hard to find misses, and haven't. Looks fixed to me.
Marking as FIXED