1243276 - Autoland pushed only part of commits and shows a wrong pusher

Status: RESOLVED FIXED

(Conduit Graveyard :: Transplant, defect)

Description

[Xidorn Quan [:xidorn] UTC+10]

Today when the tree was closed, I triggered an autoland for bug 1229437.

And just several minutes before, I saw push messages in #developers channel, and found it only pushed my part 4 & 5. Part 1 ~ 3 are missing. More weird, the pusher is not me, but :TYLin. See the push log [1].

And in MozReview interface, it shows:
> There was an error executing the Autoland request on mozilla-inbound
> Requested by xidorn
> hg error in cmd: hg push -r tip ssh://hg.mozilla.org/integration/mozilla-inbound/: pushing to ssh://hg.mozilla.org/integration/mozilla-inbound/ searching for changes no changes found 

[1] https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=04c819d845d5

Comment 1

[Mike Hommey [:glandium]]

Apparently tlin triggered an autoland in https://reviewboard.mozilla.org/r/32231/, which a) didn't land and b) ended up with half of xidorn's autoland.

Comment 2

[Mike Hommey [:glandium]]

I'm tempted to say autoland should be cut off until this is fixed.

Comment 3

[Xidorn Quan [:xidorn] UTC+10]

And it says "All the jobs passed on mozilla-inbound (2 jobs success)" for that, but appearantly almost all jobs failed.

Comment 4

[Nigel Babu [:nigelb]]

Trees have been set to approval-only to avoid more autolander landings until we can confirm the the issue is fixed or turn it off so it can be investigated.

Comment 5

[Gregory Szorc [:gps]]

I've disabled autoland in the admin ui of review board by removing the autoland url from the gecko review repo. I think it is safe to "unlock" inbound. I imagine dminor will investigate when he wakes up.

Comment 6

[Carsten Book [:Tomcat]]

(In reply to Gregory Szorc [:gps] from comment #5)
> I've disabled autoland in the admin ui of review board by removing the
> autoland url from the gecko review repo. I think it is safe to "unlock"
> inbound. I imagine dminor will investigate when he wakes up.

mozilla-inbound was changed to open after gps came online to notify us

Comment 7

[Dan Minor [:dminor]]

There was a problem with rewriting the commit messages, normally we would see only one 'base revision' message. Not sure if this is a symptom or a cause at this point.

05:36:35,444 autoland INFO initiating transplant from tree: gecko rev: d388d2e01578953ff0f27e8d76ddeefd604f7215 to destination: ssh://hg.mozilla.org/integration/mozilla-inbound/
05:36:51,977 autoland INFO base revision: 8685d4aabbc0
05:37:25,942 autoland INFO base revision: 8685d4aabbc0
05:37:55,995 autoland INFO base revision: 8685d4aabbc0
05:38:32,529 autoland INFO base revision: 8685d4aabbc0
05:38:50,43 autoland INFO transplant failed: tree: gecko rev: d388d2e01578953ff0f27e8d76ddeefd604f7215 destination: ssh://hg.mozilla.org/integration/mozilla-inbound/ error: hg error in cmd: hg push -r tip ssh://hg.mozilla.org/integration/mozilla-inbound/: pushing to ssh://hg.mozilla.org/integration/mozilla-inbound/

Comment 8

[Xidorn Quan [:xidorn] UTC+10]

BTW, I'm curious about why it is using transplant. It seems the Mercurial wiki recommends using the builtin command graft instead.

Comment 9

[Dan Minor [:dminor]]

(In reply to Xidorn Quan [:xidorn] (UTC+10) from comment #8)
> BTW, I'm curious about why it is using transplant. It seems the Mercurial
> wiki recommends using the builtin command graft instead.

We don't use 'hg transplant', the autoland module which handles landing is called that for historical reasons (at one time, there was going to be a separate webservice called 'transplant' which autoland would have used.)

Comment 10

[Dan Minor [:dminor]]

Attachment: MozReview Request: autoland: prevent partial and incorrect landings (bug 1243276) r?gps

In Bug 1243276 we ended up with a partial landing of the wrong commits. The
landing tree was closed, but due to a bug with the treestatus code we did
not find out about this until push time, which was causing us to skip the
clean up code for non-public commits. I think this missing clean up along
with the fact that we pulled and rebased the same commits multiple times
got us in to a state where 'tip' no longer referred to the commits we
wanted to push.

This patch adds a strip of non-public commits to the beginning of any
transplant attempt. In addition, we add a check to make sure that the set
of incoming commit descriptions matches the set of outgoing commit
descriptions. This should prevent us from landing incorrect commits unless
we have somehow associated commit descriptions with the wrong commits.

I tested the new code manually by changing the handling of the encoding of
commit descriptions so they did not match after rewriting, but I not sure
we can test this automatically without having races between the test code
and autoland.

Review commit: https://reviewboard.mozilla.org/r/32649/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/32649/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8712790 [details]
MozReview Request: autoland: prevent partial and incorrect landings (bug 1243276) r?gps

https://reviewboard.mozilla.org/r/32649/#review29401

I wish we could have a test for this. But the new code seems fine. And obviously existing tests didn't start failing. So LGTM.

Comment 12

[Dan Minor [:dminor]]

Landed here: https://hg.mozilla.org/hgcustom/version-control-tools/rev/2c487b8be40d

Since it's nearly end of day for me I'm going to hold off deploying and re-enabling autoland until tomorrow so I'm around in case of any further problems.

Comment 13

[Dan Minor [:dminor]]

I've deployed this. Autoland to inbound will remain disabled for now due to an unrelated problem.