1082641 - Pushes to hg.mozilla.org that return success result in commits that aren't stored on the hg.mozilla.org server

Status: NEW

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: relevant logs from the gaia-try triggering system.

The gaia-try hook creates commits and then pushes them to hg.mozilla.org/integration/gaia-try.  A couple of times in the last few months, we've hit a scenario where a commit is pushed to hg.mozilla.org with a successful return status from hg.mozilla.org, but the commit isn't present on hg.mozilla.org when checked for later on.

The code that's doing the pushing has been observed handling a failure to push correctly in the past, and it works properly 99% of the time.  It's unlikely that this is a race condition because the only thing that I know of that commits to this repository is this system and there's currently only a single node running commits.  The pushes are not forced, just running the equivalent to 'hg push ssh://hg.mozilla.org/integration/gaia-try'

The result from this is that we silently don't get tests triggered for a commit of Gaia.

An example case was '400c64d10d24', which happened today.  I've included a log from the system which creates the commits.  Kendall looked at the HG logs and found a section which might be the log of this commit.  I'm not sure what is public or not, so I'll let him add that to this bug if it's OK.

Comment 1

[Kendall Libby [:fubar] (he/him)]

Attachment: hg-push.log entries

added hg-push.log entries. first is successful push, second is failed push that returned as succeeded. only difference is missing pushkeys bit on second push.

Comment 2

[Gregory Szorc [:gps]]

pushkeys is the wire protocol command Mercurial uses to advance bookmarks and phases. If the push didn't actually push any new changesets or didn't change bookmarks or phase information, the client likely won't issue a pushkeys command.

Comment 3

[Gregory Szorc [:gps]]

Can you please provide a link to the source that is making the calls to hg that a) does the push b) polls hg to see what is available. I'd like to audit it for Mercurial appropriateness and to possibly identify root cause.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Gregory Szorc [:gps] from comment #3)
> Can you please provide a link to the source that is making the calls to hg
> that a) does the push b) polls hg to see what is available. I'd like to
> audit it for Mercurial appropriateness and to possibly identify root cause.

a) I'm using this library: https://bitbucket.org/jacob4u2/node-hg and invoking it like this https://github.com/jhford/try-server-hook/blob/master/gaia_try.js#L158

The second parameter to the callback for errors determines whether or not the error is the kind that should be retried.  The retry mechanism works and I've seen pushes fail, so the library seems to work.  The library just wraps the hg command server.

b) I don't control that code, but I think it *might* be a combination of the pushlog and http://hg.mozilla.org/build/buildbotcustom/file/6c129093459b/changes/hgpoller.py .  This issue isn't with the triggering of the tests.  Whenever commits are permanently added to hg.mozilla.org/integration/gaia-try, we get tests.  The issue is that the commits sometimes aren't being created and no tests are started.