658934 - When a build is canceled via self-serve the builder needs to clobber

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

The self-serve API lets us kill builds, but this appears to leave the slave with broken object directories at times.  We've had a bit of weirdness on m-c recently that was fixed by clobbers that I believe can be attributed to this.

The self-serve API should schedule the builder for a clobber when a build is killed to avoid any broken object directories gumming up later builds.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Alternatively, the interruption could be less brutal than SIGKILL.

Comment 2

[Justin Wood (:Callek)]

(In reply to comment #1)
> Alternatively, the interruption could be less brutal than SIGKILL.

By initiating buildbots "STOP BUILD" function, or is that internally using SIGKILL and is what you meant/what we use now?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Dustin will come along and give you the absolutely correct info, but IIRC buildbot is as brutal as SIGKILL when you use Stop Build.

Comment 4

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Yeah, being less brutal works for me too.

Comment 5

[Armen [:armenzg]]

dustin ^

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Depends on the operating system, among other things, and on the buildslave version.  Clobbering-after-stopping is by far the safer option, at any rate.

Comment 7

[Phil Ringnalda (:philor)]

I cancelled five Android builds; so far, only two of the five have burned their next builds. Of course, only two of the five have *had* next builds...

Comment 8

[Chris AtLee [:catlee]]

I chatted with Dustin about this. We think the best approach is to wrap the make invocation with steps that touch a file, and then delete the file afterwards. If the build is cancelled while compiling, the file will exist on the next build run. clobberer.py can then notice this, and run a clobber.

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

adding this to the dep tree for bug 697101 which would allow clobbers to happen with API calls.

Comment 10

[Ed Morley [:emorley]]

This bit us again for the second time in as many weeks today. Would it be possible to bump the priority of this? :-)

Comment 12

[Ed Morley [:emorley]]

The sheriffs fairly frequently have situations where we could have cancelled unnecessary jobs to reduce infra load, but don't because the "cancel all" button would cause build bustage & cancelling each (non-breaking) job is just too tedious.

Out of this, bug 666756 & bug 673246; this bug looks like the lowest hanging fruit to enable us to reduce infra load.

As such catlee, what would you think about raising the priority on this one? :-)

Comment 13

[Chris AtLee [:catlee]]

Attachment: add new API for clobberer

I'd rather not tie self-serve directly to the clobberer db, and use APIs if possible.

This patch adds a new API to clobberer where you can pass in slave-$slavename=$buildername as POST parameters, and clobberer will create the appropriate entries in the clobber_times table.

This API could then be used by the self-serve agent when cancelling a build. The flow would be something like this:
- user clicks cancel on tbpl
- request goes into self-serve, and is received by the self-serve agent
- the agent figures out which master, builder name, and build number is being cancelled. we already have this logic
- *NEW* the agent scrapes the master's build page to figure out which slave the build is running on
- *NEW* the agent uses the builder name and slave name and the new clobberer API to scheduler a clobber
- the agent cancels the build via the buildbot interface

Comment 14

[John Hopkins (:jhopkins)]

Comment on attachment 716488 [details] [diff] [review]
add new API for clobberer

Looks good.

Only one nitpick, and that's to avoid "select *" when you can and name the individual fields that are needed.  Mostly, this makes the purpose of the query more self-evident.

Comment 15

[Chris AtLee [:catlee]]

Attachment: add new API for clobberer

Comment 16

[Chris AtLee [:catlee]]

Attachment: [untested] self-serve agent support for clobbering

hoping to test this sometime this week. drive-by comments welcome!

Comment 17

[Chris AtLee [:catlee]]

I tested this, and it failed because clobberer requires authentication to trigger a clobber.

We can get around this by using real credentials for self-serve agent here, or by disabling auth for the agent (but then who shows up in the clobberer log?)

Comment 18

[Chris AtLee [:catlee]]

Just spoke with dustin, and I think we have a solution here. We can hit the same HTTP API that the build slaves use, and that doesn't require auth.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

docs updated:
  https://mana.mozilla.org/wiki/display/IT/BuildAPI
  https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=24805981

Comment 20

[Chris AtLee [:catlee]]

Attachment: clobber builds when cancelled

Comment 21

[Chris AtLee [:catlee]]

Attachment: add clobberer url to self-serve config