699838 - Set up a project branch to allow us to run several iterations for metrics

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[cmtalbert]

Stephen has determined that the number of iterations we currently
run with our talos tests is far too few to really generate solid
statistical data. He can determine how many we need to optimally
run, but to do that, he needs to generate some data with a bunch
of talos iterations.

We would like to get a project branch setup for this work, with a modified buildmaster so that it sends talos an amended commandline that indicates to talos the number of iterations it should run.

The bug to implement the command line for number of iterations in talos is bug 699512

Comment 1

[Justin Wood (:Callek)]

(In reply to Clint Talbert ( :ctalbert ) from comment #0)
> We would like to get a project branch setup for this work, with a modified
> buildmaster so that it sends talos an amended commandline that indicates to
> talos the number of iterations it should run.

this sounds like a temporary project branch from the description here. Which to me means I should strongly object to a brand new branch being created and instead advocate the use of a twig repo. And if there are no twigs available advocate a new twig be setup.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Justin Wood (:Callek) from comment #1)
> (In reply to Clint Talbert ( :ctalbert ) from comment #0)
> > We would like to get a project branch setup for this work, with a modified
> > buildmaster so that it sends talos an amended commandline that indicates to
> > talos the number of iterations it should run.
> 
> this sounds like a temporary project branch from the description here. Which
> to me means I should strongly object to a brand new branch being created and
> instead advocate the use of a twig repo. And if there are no twigs available
> advocate a new twig be setup.

Yup, project branch == twigs. I've reserved the pine branch for this work as of today.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I"m not very familiar with this project, but if the builds you're testing are arbitrarily chosen and all the interesting changes occur in the buildbot config for the test master, then a project branch isn't required. You just need to pick a build and work out the appropriate sendchanges to generate tests on demand.

Comment 4

[cmtalbert]

(In reply to Nick Thomas [:nthomas] from comment #3)
> I"m not very familiar with this project, but if the builds you're testing
> are arbitrarily chosen and all the interesting changes occur in the buildbot
> config for the test master, then a project branch isn't required. You just
> need to pick a build and work out the appropriate sendchanges to generate
> tests on demand.

That's true, but we need the buildbot to be reporting properly so that the test results end up going to pulse.  And I think that without a tree, it may not go through those steps, right?

Comment 5

[cmtalbert]

Oops, somehow my last change removed the priority on this bug, and I didn't mean to.  The dependent bug is now fixed, how do we deploy our specific talos configuration to this branch?  Is that entirely in capable releng hands or do you need something from us?  Thanks!

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Lukas Blakk [:lsblakk] from comment #2)
> (In reply to Justin Wood (:Callek) from comment #1)
> > (In reply to Clint Talbert ( :ctalbert ) from comment #0)
> > > We would like to get a project branch setup for this work, with a modified
> > > buildmaster so that it sends talos an amended commandline that indicates to
> > > talos the number of iterations it should run.
> > 
> > this sounds like a temporary project branch from the description here. Which
> > to me means I should strongly object to a brand new branch being created and
> > instead advocate the use of a twig repo. And if there are no twigs available
> > advocate a new twig be setup.
> 
> Yup, project branch == twigs. I've reserved the pine branch for this work as
> of today.

Is there anything else needed here, or can we just close this as FIXED?

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

> Is there anything else needed here, or can we just close this as FIXED?

Yes, I need to make the customizations to the pine branch so that pushes to that branch will get the special 100x talos runs.  Local testing started, patch forthcoming.

Comment 8

[Stephen Lewchuk]

Attachment: Change to logparser to monitor pine branch

This should be the only change needed for logparser to collect the data which will be generated on the pine branch.  Without anything being generated on pine I haven't been able to test this.

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: buildbot configs & project_branches patch to enable 'suites' customization for handing in a custom --sampleConfig

Today in irc I confirmed which suites need the custom runs:
12:32 #build:  slewchuk> lsblakk: chrome, chrome_mac, nochrome, tp, and svg are the testgroups that should use cycles.config

Here is the patch that allows us to customize the --sampleConfig that gets sent to PerfConfigurator. I had to deepcopy the SUITES to the BRANCHES when we look at custom talos (ie: project branches) and it's needed so that I can make changes to the '--sampleConfig' only for the branches where that is request (and not all of them) - hence the change in the .cfg files to check for BRANCHES[branch].get('suites', SUITES)

Also, there is an unfortunate issue with getting the remote-* talos suites even when mobile is not enabled, but I am not fixing that in this particular bug so I have just worked around it for the pine branch.

I will attach dump_masters so you can see that this only affects the pine branch (and disables all other tests/talos that don't need this custom cycles file).

Also in testing locally, the cycles.config file was not found so I will go confirm that that has landed or will land before this change goes in and then pine branch will be ready to go.

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: dump_masters output with the new changes

This is the dump_masters output - you can see that only pine is affected with the new --sampleConfig being sent in, and the other changes are just re-ordering of the builder name - noop.

Comment 11

[Lukas Blakk [:lsblakk] use ?needinfo]

Just checked with jmaher and we're fine to land this anytime, they will do the talos.zip deploy when they are ready, separately.  Since this change is only on the pine branch which is reserved for ateam right now, it can be active and unused and once they land cycles.config they can start pushing to pine and seeing their results (and multiple runs) right away.

Comment 12

[Armen [:armenzg]]

Comment on attachment 576238 [details] [diff] [review]
buildbot configs & project_branches patch to enable 'suites' customization for handing in a custom --sampleConfig

Review of attachment 576238 [details] [diff] [review]:
-----------------------------------------------------------------

Just two comments nits. r+ by addressing them.

::: mozilla-tests/config.py
@@ +449,3 @@
>      if branchConfig.get('talos_suites'):
> +        for suite, settings in branchConfig['talos_suites'].items():
> +            # only want the 0 or 1 for talosConfig, other customizations get put into SUITES

I know what you are trying to say with the comment but I am having trouble parsing it. I can't express in words an alternative so feel free to check-in as is.

@@ +454,5 @@
> +                # append anything new in 'suites' for a talos_suite
> +                for key, value in settings[1].items():
> +                    if suite in SUITES.keys():
> +                        BRANCHES[branch]['suites'][suite][key] += value
> +                        #SUITES[suite][key] += value

Any reason you want that comment there?

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 576238 [details] [diff] [review]
buildbot configs & project_branches patch to enable 'suites' customization for handing in a custom --sampleConfig

landed on default http://hg.mozilla.org/build/buildbot-configs/rev/d60ef4307c45

When the next reconfig happens and this goes into production builds can be triggered on Pine branch (as long as cycles.config is landed and deployed in talos.zip).

Comment 14

[Jonathan Griffin (:jgriffin)]

Comment on attachment 576053 [details] [diff] [review]
Change to logparser to monitor pine branch

I'll deploy this soon.

Comment 15

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #14)
> Comment on attachment 576053 [details] [diff] [review] [diff] [details] [review]
> Change to logparser to monitor pine branch
> 
> I'll deploy this soon.

Landed and deployed.

Comment 16

[[e-mail] (:bhearsum)]

This made it to production today.

Comment 17

[Lukas Blakk [:lsblakk] use ?needinfo]

The new talos.zip containing cycles.config is now live so please go ahead and push to the pine branch to start getting your custom metrics runs on the requested suites.