678527 - MXR not updating in > 2 weeks

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

MXR for mozilla-central hasn't updated since before the beginning of the month.

Comment 1

[Corey Shields [:cshields]]

Do you have an exact date?

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Well, I have a lower bound.

http://hg.mozilla.org/mozilla-central/pushloghtml?changeset=eee7a94c44e2 was pushed on July 28th and added browser/devtools, which is not in MXR.

Comment 3

[Jesper Kristensen]

The "hg log" and "hg blame" links points to http://hg.mozilla.org/mozilla-central/pushloghtml?changeset=982a5835fba1

Comment 4

[Mike Shaver (:shaver emeritus)]

If I write a shell script that returns a success code for a fresh-enough mozilla-central database, and a failure code for one that's too old, will someone wire it into nagios for us?

Comment 5

[Corey Shields [:cshields]]

the fix for this has more to do with bug 676432 and the fact that glimpse is not able to finish indexing..  I/O is hanging too much

We are working a -possible- issue with Dell that may be to blame for this, MXR slowness, and Tinderbox issues.  No ETA but we are working on it as we speak.

Comment 6

[Eric Shepherd [:sheppy]]

fwiw, this is something of a blocker for developer documentation work, since all our tools and processes are centered around MXR.

Comment 7

[Corey Shields [:cshields]]

(In reply to Eric Shepherd [:sheppy] from comment #6)
> fwiw, this is something of a blocker for developer documentation work, since
> all our tools and processes are centered around MXR.

We've been focused on fixing tinderbox first, since that has been something of a blocker to, well, development..

the issue behind tinderbox and mxr are the same, and data for mxr is being moved to trays that are not oversubscribed as we speak.

Comment 8

[Eric Shepherd [:sheppy]]

(In reply to Corey Shields [:cshields] from comment #7)
> (In reply to Eric Shepherd [:sheppy] from comment #6)
> > fwiw, this is something of a blocker for developer documentation work, since
> > all our tools and processes are centered around MXR.
> 
> We've been focused on fixing tinderbox first, since that has been something
> of a blocker to, well, development..

Ah, I hadn't been aware that tinderbox is broken (I don't use it personally). Obviously that's the priority, yes.

Comment 9

[Corey Shields [:cshields]]

Yeah..  I haven't been updating everything too clearly either (sorry - we have been a couple of people down and I've been dealing with this issue while traveling to do interviews and new hire training)..  I'll toss an update into 676432 since that is the more relevant to the root problem.

Comment 10

[Corey Shields [:cshields]]

Is this still out of date?  How do I test?

I don't see the cron jobs stuck per se but if we are out of date still I will kill them all off and this should resolve itself within a day.

Comment 11

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We're still out of date.  My test has been to load http://mxr.mozilla.org/mozilla-central/source/browser/ and see if there's a 'devtools' directory in there.

Comment 12

[Corey Shields [:cshields]]

I have run the index updates by hand (both the nightly and the */4 hr cron) and although they succeed, devtools is not showing up.

Going to ask justdave for help on this one, I believe he has more tribal knowledge on MXR.

Comment 13

[Dave Miller [:justdave]]

[root@dm-webtools04 mozilla-central]# cat cvs.log
Thu Aug 18 08:55:56 PDT 2011
/var/www/webtools/mxr
pulling from http://hg.mozilla.org/mozilla-central
searching for changes
abort: abandoned transaction found - run hg recover!
/data/mxr-data/mozilla-central/mozilla-central
982a5835fba1
Thu Aug 18 08:56:53 PDT 2011

So it's been stuck like that for who knows how long...
Running recover on it now.  I'll grep the logs for all the trees to see if this happened to any of the others.

Comment 14

[Dave Miller [:justdave]]

[root@dm-webtools04 mozilla-central]# hg recover 
rolling back interrupted transaction
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
76980 files, 73041 changesets, 335599 total revisions

[root@dm-webtools04 mxr-data]# find . -maxdepth 2 -name cvs.log -exec grep -q 'hg recover' {} \; -print
./mozilla-central/cvs.log

And that appears to have been the only one.

I'm manually updating mozilla-central right now, should be up-to-date in an hour or so.

Comment 15

[Dave Miller [:justdave]]

http://mxr.mozilla.org/mozilla-central/source/browser/ is now up-to-date.

Comment 16

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Dave Miller [:justdave] from comment #15)
> http://mxr.mozilla.org/mozilla-central/source/browser/ is now up-to-date.

You, sir, are my hero.