372342 - Bonsai diffs are broken for some check-ins

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Adam Guthrie]

I realize that Bonsai diffs do now show up for check-ins that were made during the last 15 minutes, but this issue is different.

Steps to reproduce:
1. Go to http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2007-03-01+00%3A00&maxdate=2007-03-01+23%3A59&cvsroot=%2Fcvsroot
2. Click on the "1.120" link after the check-in to 	mozilla/extensions/irc/xul/content/commands.js.

I get taken to the diff page, but no diff is shown. The same thing happens with other diffs on the page: e.g. bz's check-in to nsRange.cpp.

Comment 1

[Dave Miller [:justdave]]

there were stuck cvs rsync locks, I cleared them last night, but didn't know there was a bug here.

Comment 2

[Dave Miller [:justdave]]

Should be working as of 10pm or so last night.

Comment 3

[Dave Miller [:justdave]]

Ah, but it came back again since then...  locks stuck as of 8:50am again this morning.  Aravind cleared the locks again but we need to investigate why this is suddenly happening frequently...

Comment 4

[J. Paul Reed [:preed]]

(In reply to comment #3)
> Ah, but it came back again since then...  locks stuck as of 8:50am again this
> morning.  Aravind cleared the locks again but we need to investigate why this
> is suddenly happening frequently...

Whose locks are they?

Comment 5

[Dave Miller [:justdave]]

bonsai needs a local copy of cvs to perform rcs operations on.  So there's a psuedo-cvs-mirror on bonsai's server.  The locks for the rsync jobs keeping that mirror up to date are what keep getting stuck.

Comment 7

[Dave Miller [:justdave]]

We appeared to be hitting connection limits trying to run the rsync, and the lockfile would get stuck when it hit one.  I've bumped the connection limits on the cvs rsync server a day or two ago, and this should help.  I haven't seen any recurrances since then.

Comment 8

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Is this what is causing diffs that appear to contain no changes when following a bonsai revision link today (and files appearing to be a revision older than HEAD when following the filename link)?

E.g. http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&subdir=mozilla/modules/libpref/src/init&command=DIFF_FRAMESET&file=all.js&rev1=3.665&rev2=3.666&root=/cvsroot

Comment 9

[Dave Miller [:justdave]]

more than likely.  The generated diffs will always be 10 or 15 minutes behind, because of the rsync.  It rsyncs from cvs-mirror every 5 minutes, cvs-mirror rsyncs from cvs every 5 minutes, so depending when within that 5 minute window the commit happens, it will always take between 5 and 15 minutes for the data behind it to make it to bonsai.  The emails from cvs to bonsai which contain the commit message generally move quicker than that.

Comment 10

[Adam Guthrie]

I still don't think this is fixed. I'm still getting broken diffs for check-ins made earlier this afternoon (PST).

Comment 11

[Dave Miller [:justdave]]

The rsync locks have been getting stuck daily just after 5am for the last week or so for some reason.  Right now, the cvs-mirror replication scripts (which were copied verbatim to everyplace bonsai has been) do a clobber rsync of the cvs repo every day at 5am.  I changed the scripts Wednesday night on bonsai and bonsai-stage to do the clobber at 6am instead of 5am (an hour after the clobber happens on cvs-mirror so they don't run into each other).  It hasn't stuck again in the last two days, so that must have been the issue.

Comment 13

[Dave Miller [:justdave]]

OK, this is obviously still happening.

Comment 14

[Dave Miller [:justdave]]

The timing for when this all started happens to coincide nicely with when cvs-mirror was switched over to using the gslb load balancing, so punting to mrz to figure out what's up with the network...

Comment 15

[Dave Miller [:justdave]]

dm-webtools02's rsync died at 6:13am on 3/10.
dm-webtools01's rsync died at 9:03am on 3/10.

The clobber rsyncs were disabled on the afternoon on the 9th, so those are no longer running at all, we just have the normal rsyncs going now.

strace on the stuck process shows it stuck in a loop select()ing filehandle 4 (which doesn't tell me much except that a network connection probably stalled and didn't disconnect).

Comment 16

[Dave Miller [:justdave]]

select(4, NULL, [3], [3], {60, 0})      = 0 (Timeout)
select(4, NULL, [3], [3], {60, 0})      = 0 (Timeout)
select(4, NULL, [3], [3], {60, 0})      = 0 (Timeout)

Comment 17

[matthew zeier [:mrz]]

(In reply to comment #14)
> The timing for when this all started happens to coincide nicely with when
> cvs-mirror was switched over to using the gslb load balancing, so punting to
> mrz to figure out what's up with the network...

We switched off vegas weeks before moving to GSLB (just last week) - is it just since GSLB or since going to dm-cvs02, behind the Netscaler?

Comment 18

[matthew zeier [:mrz]]

Dave, any recurrence?

Comment 19

[Dave Miller [:justdave]]

No, this appears to have been resolved by dropping the local IP for cvs-mirror into the hosts file on both bonsai machines.  Having to go through the netscaler on the way to cvs-mirror was apparently killing it. (which is a separate bug I suppose)