737865 - hg changelog and manifest updates are becoming visible to the hgweb server before updates to the file log

Status: RESOLVED WORKSFORME

(Developer Services :: General, task)

Description

[(no longer active)]

I am getting consistent repository corruptions when I try to pull from mozilla-central.  I filed this as a bug with Mercurial, but the developers there believe that this is being caused by us running a script on the server side which causes the changelog and manifest updates to be exposed to the hgweb server before the updates to the file log, which makes pulling fail if you happen to pull after the changelog and manifest have been updated but before the file log updates are visible.

I'm filing this bug for investigation of this problem on our side.  The hg bug is http://mercurial.selenic.com/bts/issue3324.

Comment 1

[Ben Kero [:bkero]]

After looking at the mercurial bug linked in here, they seem to have fixed this issue.  An update of Mercurial should solve this problem.

Comment 2

[(no longer active)]

I thought that the mercurial folks believed that this is a bug on our side?

Comment 3

[Ben Kero [:bkero]]

The last comment on the bug is:


"This was fixed in http://selenic.com/hg/rev/5d75eb8568d1 , probably by mistake.", which leads me to believe that it was a problem on their side that got fixed.  Do you not agree?

Comment 4

[(no longer active)]

(In reply to comment #3)
> The last comment on the bug is:
> 
> "This was fixed in http://selenic.com/hg/rev/5d75eb8568d1 , probably by
> mistake.", which leads me to believe that it was a problem on their side that
> got fixed.  Do you not agree?

That is not the same bug (it was not filed by me).  I don't know if the original bug is still accessible somewhere...

Comment 5

[Shyam Mani [:fox2mike]]

Ehsan, still an issue?

Comment 6

[(no longer active)]

(In reply to comment #5)
> Ehsan, still an issue?

I do get corrupted repositories from time to time, yes.  But this happens much less frequently these days.

Comment 7

[John Schoenick [:johns]]

My hg <-> git mirror gets this corruption a few times per month. It's pretty bad that poorly timed hg pulls can corrupt a user's repository -- can we look in to this?

Why mercurial would allow a bad pull to corrupt your repository is another issue...

Comment 8

[Ben Kero [:bkero]]

Mercurial was not designed to be served over NFS. However they've been very accommodating in terms of accepting problems we run into that might have a root cause of hosting a repository on an NFS share.

My hope is that this is a race condition that will go away as we move to using local storage for hg web heads.

Comment 9

[Ben Kero [:bkero]]

@ehsan Could you find the bug you have with mercurial so we can figure out if this issue still exists? The bug number you reference earlier seems to have been fixed.

Comment 10

[(no longer active)]

(In reply to comment #9)
> @ehsan Could you find the bug you have with mercurial so we can figure out if
> this issue still exists? The bug number you reference earlier seems to have
> been fixed.

IIRC Mercurial changed their issue management software after I filed that issue, so I'm not sure if the old bug is still accessible anywhere (I don't have a working link to it any more.)  But if we deploy a new mercurial on the server, I can just increase the frequency of my scripts and see if the issue starts to happen again.  (FWIW I have not seen it for a while now)

Comment 11

[Gregory Szorc [:gps]]

My money is on this going away after moving off NFS.

Comment 12

[Dirkjan Ochtman (:djc)]

Here's the original bug Ehsan filed, in the new bug tracker:

http://bz.selenic.com/show_bug.cgi?id=3325

Comment 13

[Ben Kero [:bkero]]

Is this ever seen anymore? We're thinking this got solved from the NFS migration, but we'd like some confirmation.

Comment 14

[(no longer active)]

(In reply to Ben Kero [:bkero] from comment #13)
> Is this ever seen anymore? We're thinking this got solved from the NFS
> migration, but we'd like some confirmation.

I have long stopped maintaining my git mirror, so I haven't seen it recently, but that's because I have not been looking.  :-)

Comment 15

[Gregory Szorc [:gps]]

My agents doing automatic mirroring haven't seen any issues lately.

Let's call this resolved until proven otherwise.