477056 - content sync from san jose to amsterdam failing (planet.m.o out of date)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Pascal Chevrel:pascalc]

I blogged several hours ago and it doesn't appear on planet, the last blog post syndicated is like 15 hours old so I guess there is a problem syndicating content.

Comment 2

[Samuel Sidler (old account; do not CC)]

This belongs in Server Ops.

Comment 3

[Robert Accettura [:raccettura]]

Interestingly it seems like just that feed.

Comment 4

[Wladimir Palant]

(In reply to comment #3)
> Interestingly it seems like just that feed.

Which feed? The page says that last update is "February 04, 2009 08:01 PM" which is more than 24 hours ago now. My blog post doesn't show up either.

Comment 5

[Robert Accettura [:raccettura]]

A little digging is showing that it doesn't like lpsolit for some reason (bad character)... I removed those two feeds.  Lets see if that gets things going.

[http://lpsolit.wordpress.com/category/bugzilla/feed]
[http://lpsolit.wordpress.com/category/mozilla/feed]

Comment 6

[Robert Accettura [:raccettura]]

We might need someone from server ops to look at this.  No errors for the past 30 min, but doesn't look like it's updating either.

Comment 7

[Robert Accettura [:raccettura]]

I think that did the trick.  It looks like things are updating now.  

lpsolit feeds are still removed for the moment.  I'm hoping to look at that this weekend as I've been sick the past few days.

ERROR:planet.runner:Error processing http://lpsolit.wordpress.com/category/bugzilla/feed
ERROR:planet.runner:UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 220: ordinal not in range(128)
ERROR:planet.runner:  File "/data-local/planet.mozilla.org/venus/planet/spider.py", line 312, in httpThread
    (resp, content) = h.request(idna, 'GET', headers=headers)
ERROR:planet.runner:  File "/data-local/planet.mozilla.org/venus/planet/vendor/httplib2/__init__.py", line 874, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
ERROR:planet.runner:  File "/data-local/planet.mozilla.org/venus/planet/vendor/httplib2/__init__.py", line 688, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
ERROR:planet.runner:  File "/data-local/planet.mozilla.org/venus/planet/vendor/httplib2/__init__.py", line 659, in _conn_request
    conn.request(method, request_uri, body, headers)
ERROR:planet.runner:  File "/usr/lib64/python2.4/httplib.py", line 810, in request
    self._send_request(method, url, body, headers)
ERROR:planet.runner:  File "/usr/lib64/python2.4/httplib.py", line 833, in _send_request
    self.endheaders()
ERROR:planet.runner:  File "/usr/lib64/python2.4/httplib.py", line 804, in endheaders
    self._send_output()
ERROR:planet.runner:  File "/usr/lib64/python2.4/httplib.py", line 685, in _send_output
    self.send(msg)
ERROR:planet.runner:  File "/usr/lib64/python2.4/httplib.py", line 664, in send
    self.sock.sendall(str)
ERROR:planet.runner:  File "<string>", line 1, in sendall
ERROR:planet.runner:Error 500 while updating feed http://lpsolit.wordpress.com/category/bugzilla/feed

Comment 8

[Dave Miller [:justdave]]

fyi, we had a stuck lock in the webhead push for the static cluster, the lock got kicked about an hour ago.  it was 23 hours old at the time.  So planet was probably updating, and the updated html wasn't making it to the web servers.

Comment 9

[Robert Accettura [:raccettura]]

That would explain a lot... the error above generally doesn't cause planet to stop updating, it just causes that feed to be ignored during that polling.

Lets leave things for the night, I'll likely add those feeds back tomorrow or this weekend.

Comment 10

[matthew zeier [:mrz]]

Best I can tell this is fixed.  Mitchell's latest post is on pmo so it's obviously updating.

Comment 11

[Wladimir Palant]

No, not fixed. There was one update in the morning yesterday ("February 06, 2009 08:31 AM") - nothing since then. My blog from yesterday didn't show up.

Comment 12

[matthew zeier [:mrz]]

Only errors I'm seeing are:

ERROR:planet:Error 404 while updating feed <http://blog.mozilla.com/wendy/feed/>

and I am seeing current feeds pulled in and I see one from you titled "Five wrong reasons to use eval() in an extension".

Comment 13

[Wladimir Palant]

Unfortunately, that blog post still didn't make it to the web server.

Comment 14

[matthew zeier [:mrz]]

(In reply to comment #13)
> Unfortunately, that blog post still didn't make it to the web server.

(In reply to comment #12)
> and I am seeing current feeds pulled in and I see one from you titled "Five
> wrong reasons to use eval() in an extension".


This one didn't make it to planet?  I mentioned it because I actually saw it on planet.

When you 'ping planet.mozilla.org', what IP address to you get back?

Comment 15

[Wladimir Palant]

PMO is 63.245.213.14 for me - and most recent blog post is still http://starkravingfinkle.org/blog/2009/02/dom-inspector-for-fennec/ from yesterday morning.

Comment 16

[Robert Accettura [:raccettura]]

FWIW I added the above 2 feeds back in last night.  It's still updating for me.

I think it's pretty safe to say this is an operational problem rather.  mrz to the rescue.

Comment 17

[matthew zeier [:mrz]]

Last file push from San Jose to Amsterdam was on 02/06 according to what nladm01's version of pmo has (ignoring file timestamp but looking at the html). 

[root@mradm02 planet_webroot]# ls -l index.html 
-rw-r--r-- 1 root root 283533 Feb  7 09:46 index.html

root@nladm01 (/data/static/www/planet.mozilla.org/planet_webroot/) 37> ls -l index.html 
-rw-r--r--  1 root root 278551 Feb  4 12:16 index.html

Comment 18

[matthew zeier [:mrz]]

oremj, looks like this pushes in series and was stalled since yesterday pushing to cn.  Can that be parallized?

Comment 19

[matthew zeier [:mrz]]

btw, running manually.

Comment 20

[Reed Loden [:reed]]

(In reply to comment #12)
> Only errors I'm seeing are:
> 
> ERROR:planet:Error 404 while updating feed
> <http://blog.mozilla.com/wendy/feed/>

That's planet.mozillaonline.com, not planet.mozilla.org.

Comment 21

[Jeremy Orem [:oremj]]

They will run in parallel and not block on each other now.

Comment 22

[Colin Ogilvie [:cso]]

I'm still seeing different content on planet.mozilla.org depending on whether I check from my machine (UK based):

[colin@lamda ~ $] host planet.mozilla.org
planet.mozilla.org is an alias for www-mozilla-org.geo.mozilla.com.
www-mozilla-org.geo.mozilla.com is an alias for www-mozilla-org.glb.mozilla.com.
www-mozilla-org.glb.mozilla.com has address 63.245.213.14


versus landfill:

[colin@landfill ~]$ host planet.mozilla.org
planet.mozilla.org is an alias for www-mozilla-org.geo.mozilla.com.
www-mozilla-org.geo.mozilla.com is an alias for www-mozilla-org.glb.mozilla.com.
www-mozilla-org.glb.mozilla.com has address 63.245.209.11

This would still appear to be a problem...

Comment 23

[Wladimir Palant]

Yes, the content on 63.245.213.14 is still outdated (two days old). Right now I have 63.245.209.11 hardcoded in my hosts file to see current PMO content.

Comment 24

[Robert Accettura [:raccettura]]

It was updating for me, it no longer is. 63.245.209.11 seems outdated as well.

mrz, oremj: can you guys take another look at this?

Comment 25

[Wladimir Palant]

No, 63.245.209.11 is certainly updating every half an hour. Note that you need to restart the browser every time you change the hosts file, otherwise changes will not be picked up.

Comment 26

[Pascal Chevrel:pascalc]

We still get February 8 content in France (tested from 3 different ISPs)

Comment 27

[Robert Accettura [:raccettura]]

I think what I was seeing before was squid cache on my end.  Sorry.

Comment 28

[Jeremy Orem [:oremj]]

The admin sync script has been fixed and should solve this problem.

Comment 29

[Wladimir Palant]

Confirmed, seems to be updating correctly (however, PMO resolves to 63.245.213.12 rather than 63.245.213.14 for me now).

Comment 30

[matthew zeier [:mrz]]

> Confirmed, seems to be updating correctly (however, PMO resolves to
> 63.245.213.12 rather than 63.245.213.14 for me now).

That's normal - there are two load balancers out there.