202060 - CVS-committed changes to mozilla.org content not becoming visible

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Alex Bishop]

Late on Friday night, I used Doctor to check in the latest status update
(http://www.mozilla.org/status/2003-04-11.html) and update
http://www.mozilla.org/status/ and http://www.mozilla.org/ with links to it.
However, none of these changes have become visible yet, almost 72 hours later.

Bonsai tells me that http://www.mozilla.org/news.html was updated on Friday at
00:42 and that change is visible, so the breakage probably occurred some time
later that day.

Comment 1

[Dave Miller [:justdave]]

-> Server operations

Comment 2

[Dave Miller [:justdave]]

Looks like the /d partition was full on gila...

Myk just cleared some space on it.  However, that's not where the docroot is, so
it's not clear at this time whether that'll solve the problem.

Comment 3

[Myk Melez [:myk] [@mykmelez]]

Attachment: patch v1: fix for problem

Found it; NS-query.pat got backed out, but munge.pl didn't get updated to stop
looking for it.  Heh, that file isn't even from the last search system but
rather two systems ago.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

Ok, fixed.  Brant, please check for dependencies in the future before deleting
non-static files.

Checking in tools/munge.pl;
/cvsroot/mozilla-org/tools/munge.pl,v  <--  munge.pl
new revision: 1.21; previous revision: 1.20
done

Also removed:

search/html/HTML-tocend.pat
search/html/HTML-tocrec.pat
search/html/HTML-tocstart.pat

Comment 5

[Myk Melez [:myk] [@mykmelez]]

Urk.

Comment 6

[Brant Gurganus]

Woops, sorry.  For my reference, which scripts are actually getting run on the
server side.  There are some that don't seem to be used like check.pl while
others are used like munge.pl apparently.

VERIFIED since status updates appear updated.

Comment 7

[Myk Melez [:myk] [@mykmelez]]

mozilla-org/tools/dostage is run, plus whatever it says in mozilla-org/Makefile.

Comment 8

[Boris Zbarsky [:bzbarsky]]

Reopening.  The footer did update, but I checked in some changes last night
("Apr 17 23:11") that still has not propagated.  Compare
http://www.mozilla.org/catalog/libraries/uriloader/ and
http://www.mozilla.org/webtools/bonsai/cvslog.cgi?file=mozilla-org/html/catalog/libraries/uriloader/index.html&rev=&root=/cvsroot/

Comment 9

[Alex Bishop]

Boris's changes seem to be visible now. However, yesterdays's status update,
checked in at 23:47, hasn't appeared yet (the links to it on
http://www.mozilla.org/status/ and the front page aren't there either).

So something's still up.

Comment 10

[Brant Gurganus]

Its not me this time.  I haven't changed anything else.

Comment 11

[Myk Melez [:myk] [@mykmelez]]

The problem is that the cronjob isn't running for some reason.  I'm not sure why.

Comment 12

[Myk Melez [:myk] [@mykmelez]]

cron is having problems.  It reports "! c queue max run limit reached" over and
over again.  This is apparently because it is hitting its 100 job limit, and
such problems are usually caused by jobs that refuse to quit.  Looking at the
logs for the last few minutes, the culprit seems to be one or both of the following:

>  CMD: cd /opt/newsbot ; ./newsbot.sh > /dev/null 2>&1
>  root 12086 c Mon Apr 21 07:25:00 2003
>  CMD: cd /e/webtools/bonsai ; ./processqueue.pl

I've disabled both.  This should fix the problem at the cost of breaking
whatever it is those two scripts were doing.  newsbot we can probably ignore;
it's dead I think.  The other one might cause bonsai not to update; we'll just
have to keep an eye out for that.

Comment 13

[Myk Melez [:myk] [@mykmelez]]

Note that "/etc/init.d/cron stop" worked, but "/etc/init.d/cron start" didn't. 
I had to run "/usr/sbin/cron" by hand to get it started.

Comment 14

[Dave Miller [:justdave]]

That usually means the existing one didn't quit.  The init scripts check, and
will refuse to start if there's already a running process by that name.  I
usually have the same problem with Apache on mothra...  you have to kill it by
hand first before restarting it.

Comment 15

[Myk Melez [:myk] [@mykmelez]]

I don't think that was the problem this time, since I checked to see if cron was
running via ps and also via the ps command /etc/init.d/cron uses.