550494 - planet showing two-day old content (stale git lock, maybe?)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Reed Loden [:reed]]

http://planet.mozilla.org/ is currently showing me content from two days ago (Wednesday, 3 March 2010 :: 3:30pm). stale git lock on one or more of the webheads, maybe?

Comment 1

[Shyam Mani [:fox2mike]]

Hmm, poking around.

The cron jobs have been running fine, no issues there.

Comment 2

[Shyam Mani [:fox2mike]]

Seems okay now, I just manually pushed the content out again. Nothing was wrong, as such. 

I'll try to poke around further and see if something failed. Re-open if this falls back again.

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

After updating "normally" all morning, it's back to showing Wednesday's content (dwitte on cookies) again :(

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #3)
> After updating "normally" all morning, it's back to showing Wednesday's content
> (dwitte on cookies) again :(

This righted itself sometime this afternoon; however, right now, we're back to to showing Wednesday's content (dwitte on cookies) again :(

Comment 5

[Phong Tran [:phong]]

I'm seeing contents from today.

Comment 6

[Reed Loden [:reed]]

(In reply to comment #5)
> I'm seeing contents from today.

One (or more) of the webheads is likely out of sync... I'm seeing Wednesday's content here.

Comment 7

[Phong Tran [:phong]]

i manually pushed out the content again.

Comment 8

[Reed Loden [:reed]]

http://planet.mozilla.org/?nocache222 is still showing the old content... Did the sync finish yet?

Comment 9

[Reed Loden [:reed]]

Still seeing old content....

Comment 10

[Phong Tran [:phong]]

just cleared NS cache.

Comment 11

[Reed Loden [:reed]]

(In reply to comment #10)
> just cleared NS cache.

It's not a caching problem... There has to be at least one webhead with old content. Hit each of the webheads directly using curl and see which one has old content.

Comment 13

[Reed Loden [:reed]]

This needs to be fixed now, please.

Comment 14

[Dave Miller [:justdave]]

All seven webheads are showing the current content when I hit them directly.

Comment 15

[Dave Miller [:justdave]]

I'm getting current content when I hit the live site, too?

Comment 16

[Robert Accettura [:raccettura]]

Latest I see is:

March 06, 2010
Wladimir Palant — Linux New Media Award for Adblock Plus

That seems right.

Comment 17

[Dave Miller [:justdave]]

OK, found the problem.  This content is also being served out of the new Phoenix datacenter, and the Phoenix webheads are both out-of-date and not included in the list of webheads that mradm02 knows about.  Trying to figure out how

Comment 18

[Dave Miller [:justdave]]

er, trying to figure out how to resolve it now, as it apparently is syncing content but there's a docroot mismatch or something.

Comment 19

[Dave Miller [:justdave]]

Had this comment typed several hours ago and forgot to submit it...

We apparently have problems with the way the webheads are set up in Phoenix right now.  Going to take a while to figure it out, so I've removed Phoenix from the pool for this site for now.  (And coincidentally www.mozilla.org as well, since they share an IP address)

Comment 20

[Robert Accettura [:raccettura]]

Thanks for the update Dave.

Comment 21

[Jeremy Orem [:oremj]]

Content sync -> phx has been fixed.

Comment 22

[Dave Miller [:justdave]]

per comment 21.