1066636 - Add generic builders to try

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

>> I think the basic idea is that we create one buildbot builder per
>> platform, eg "generic-win32-try". The builder implementation would be
>> a scriptfactory that runs a simple bootstrapping script that looks at
>> the build properties to determine what actions to take. From there
>> it's up to you!

The scope of this has changed a bit from what I originally requested, however, this is more generic and will get us much more.

This is how I envision this and could be rather different to what you think, so feel free to throw a whole different approach.

What I see is:
1) We add a build builder per try build pool rather than platform
* linux64, win64, macosx
* I assume we won't have different builders per sub-pool (e.g. spot, ec2, ix, scm)

2) We add a test builder per try test platform
* generic-win7-try, generic-win8-try and on

3) The bootstrapping script would listen to a defined set of build properties. Users of this service can trigger these builders by hitting the arbitrary job API.

What properties should such bootstrapping script listen to?
One approach could to be to listen to one single property like "url_to_script_to_run". That would allow the bootstrap script to download a script and run it.

A further optimization of this is if we could specify the binary calling the script. For instance, we can try newer versions of python. I believe we should keep this out of scope since we can easily extend this in the future if we wanted to.

Use cases of this could be:
* The user wants to adjust an existing specific suite
* The user wants to try some crazy stuff
* The user watches certain external events and triggers a job

Unfortunately, there is an issue with wild ideas, that is, we can't display it on tbpl unless there is a try push and a revision to associate it with.
I think the developer could upload the logs somewhere and use a pulse listener to notify him of the job finishing and where to grab the results.
I think a tool could be written for developers to have a place to listen for their own results. I believe this should be out of scope.

A further extension of this would be to allow, from treeherder, to load up a page with a script to reproduce exactly what happened on a job + having the ability to edit it + triggering it.

Comment 1

[Armen [:armenzg]]

I'm leaning to focus only on python scripts as well as specifying a requirements.txt file to download as well.

Comment 2

[James Graham [:jgraham]]

Is there a more limited/achievable version of this where we have one "generic builder" job per platform that is only enabled on try (and not by default). This looks in the tree for a) a custom mozharness repo to use and b) the script to use in that repo. Then results are just reported in the normal way. I admit I haven't thought this through very clearly, but it seems like piggybacking on the custom mozharness stuff is the most natural way to get this.

Comment 3

[Rail Aliiev [:rail]]

I think you will be able to schedule any builds/tests from the tree with taskcluster. It's not worth investing into a buildbot-based solution.