592564 - Create a semi-automated way to merge tryserver pushes to mozilla-central

Status: RESOLVED DUPLICATE of bug 657828

(Release Engineering :: General, defect, P5)

Description

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Some of us were talking on IRC today about things we could do to make development less painful.  One of the ideas was to come up with a way to get patches that pass tryserver into mozilla-central easier.  The basic concept is this:

1. Developer pushes a set of patches to try
2. Try builds and runs, results come out on TBPL.
3. Developer/philor/etc star known orange test failures
4. If all of the test failures are starred, tryserver provides some way (maybe an email, special ldap protected link, etc) to say "please check this in for me"
5. Tryserver merges the patches pushed to the tip of m-c and pushes.

The tricky part is probably #5.  If the merge has any conflicts at all the process should just abort IMO.  The other thing to deal with is scheduling of the actual pushes.  That I don't have any easy answers for, but I'm sure we can come up with something.

Comment 1

[Justin Lebar (not reading bugmail)]

Worth noting is how Webkit does this: They put a patch up on Bugzilla and flag it checkin-needed.  Then the checkin bot comes by, pulls the patch, presumably runs some tests, and checks it in.

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

So, the tricky bit here is that these automated pushes need to respect the status of the tree (not push when it's CLOSED, when there's unstarred orange/red, etc).  The easiest way to do this is to have the "queue" of automated pushes that are manually released by a sheriff/somebody human.

Another, and IMO better solution, is if/when we can grab all of the relevant data (tree status, whether or not there are unstarred test failures, and time since the last push) is to automatically push when it's been N minutes since the last push and there are no unexplained failures.  We could hook this up to firebot to warn #developers beforehand "five minutes before an automated merge from tryserver" and possibly provide a way to suppress it if someone is queueing up to push.

I don't think we should attempt much in the way of error recovery.  If a push happens and it causes test failures or breaks I think the system should just wait for a human to come along and clean up.

Comment 3

[(not currently active) Ted Mielczarek]

(In reply to comment #2)
> So, the tricky bit here is that these automated pushes need to respect the
> status of the tree (not push when it's CLOSED, when there's unstarred
> orange/red, etc).  The easiest way to do this is to have the "queue" of
> automated pushes that are manually released by a sheriff/somebody human.

The tree status could be picked up the same way the tree closure hook does, but I agree that having this be human-controlled is probably the best first step. The sheriff should have a webapp with a list of checkins, and be able to click to land them.

Comment 4

[[e-mail] (:bhearsum)]

That sounds like a really great way to try this out. I think Developers are probably more in tune with the sort of semantics and rules that an auto landing system would need, and this would be a pretty safe way to explore and tune things.

Based on recent RelEng meetings I don't think any one of us is going to be able to pick this up right away, but that shouldn't stop anybody else.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We probably could implement a human gated webapp on top of the existing infrastructure as a prototype with only the existing releng data.

Comment 6

[[e-mail] (:bhearsum)]

I think Lukas has been working on this...possibly a forward dupe of bug 657828.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Most definitely a dupe of bug 657828 - you can follow the project here: https://wiki.mozilla.org/BugzillaAutoLanding

Try automated landings via bugzilla are almost ready for widespread use, in Q1 there will be more work on landing to branches after a successful try run (however that criteria is gauged per branch) to other branches.

Not only will there be a way to do this through Bugzilla but there is also planning for an API so that command line push-to-try-then-to-$branches would be possible once the criteria are met (tree status, review status, LDAP permissions on repos requested for landing).