566038 - planet (sporadically) showing three-day old content (take 3!)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

http://planet.mozilla.org/ is currently showing me content from three days ago (Tuesday, 11 May 2010 :: 18:46:27 2010).

This happened once before this afternoon, and it was "fixed" with a reload each time.

All of this has happened before (see bug 560433 and bug 550494)…

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Just hit this again; "Last update: Tue May 11 18:46:27 2010", but a reload got me "Last update: Sat May 15 01:45:45 2010"

Comment 2

[Shyam Mani [:fox2mike]]

So how is this an IT issue? It could be a local cache issue on your end, because it gets resolved by the reload?

OTOH, if it is an issue on our end, can you capture headers the next time? Full headers, will have the backend server that's serving out old content and will make it infinitely more easier for us to troubleshoot :)

Also, where are you located? USA? Can you run a host planet.mozilla.org and attach the output here? I'm hitting the cache servers in Amsterdam and see upto date content (from the 17th) which means the origin stuff in SJC is fine.

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #2)
> So how is this an IT issue?

Because that's where bug 550494 ended up and I surmised it was a similar problem (and, in the case of bug 560433 which seemed to resolve itself eventually, many others also saw it).

> It could be a local cache issue on your end,
> because it gets resolved by the reload?

Not browser cache, because the previous views had all been the current content, i.e., I viewed planet, saw content from that day, closed the tab, waited an hour or so, viewed planet, saw three-day-old content, reloaded, and again saw content from that day.

I also tend to think it's not ISP-related, because I had the problem on two different ISPs (Bellsouth/AT&T and Earthlink).

> OTOH, if it is an issue on our end, can you capture headers the next time? Full
> headers, will have the backend server that's serving out old content and will
> make it infinitely more easier for us to troubleshoot :)

Sure, but how would I do that (I'm not a network whiz :) )?  Also, since a reload fixes it, won't trying to get headers after I see that I'm getting stale content return good headers?

> Also, where are you located? USA? Can you run a host planet.mozilla.org and
> attach the output here? I'm hitting the cache servers in Amsterdam and see upto
> date content (from the 17th) which means the origin stuff in SJC is fine.

Outside of Atlanta.  Right now I'm on the Bellsouth/AT&T network (and seeing today's planet content):

[Qalaat-Samaan:~] smokey% host planet.mozilla.org
planet.mozilla.org is an alias for www-mozilla-org.geo.mozilla.com.
www-mozilla-org.geo.mozilla.com is an alias for www-mozilla-org.glb.mozilla.net.
www-mozilla-org.glb.mozilla.net has address 63.245.217.21
;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached

I suppose it could have been some intermittent weirdness on the 14th, too; I haven't seen it again since then.

Comment 4

[Shyam Mani [:fox2mike]]

(In reply to comment #3)

> Sure, but how would I do that (I'm not a network whiz :) )?  Also, since a
> reload fixes it, won't trying to get headers after I see that I'm getting stale
> content return good headers?

The Live HTTP headers addon to Firefox is a good start. Yes, you'll get good headers too, but if you can get both bad and good, that'll help as well.
 
> www-mozilla-org.glb.mozilla.net has address 63.245.217.21

That means you're probably hitting stuff in phx

Comment 5

[matthew zeier [:mrz]]

Need to verify content sync to Phoenix is working.  Bumping sev & passing to oremj.

Comment 6

[Jeremy Orem [:oremj]]

The scripts were using git-cmd and git was upgraded automatically by puppet, so that form was no longer valid.  Changed the update script to use "git cmd" instead which is valid.

Comment 7

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #4)

> The Live HTTP headers addon to Firefox is a good start. Yes, you'll get good
> headers too, but if you can get both bad and good, that'll help as well.

Not so useful for Camino, though :(

Anyway, glad this got sorted out!