1621765 - Further improvements to SETA

Status: RESOLVED WONTFIX

(Testing :: General, enhancement, P3)

Description

[Bob Clary [:bc] (inactive)]

There have been a lot of good ideas about how to go forward with seta. I'll write down the ones I'm familiar with and we can fill in more as we discuss seta's future.

tomprince brought up the idea that we should use taskcluster to provide more data to seta on treeherder regarding the tasks to consider. The idea was to either include additional data in the runnable-jobs.json or to use new artifact which would only used by seta and thus not coupled to runnable-jobs. I believe the consensus was to introduce a new artifact. I suppose we could add optimization stanzas to the tasks via the yml files.

ahal: Are you already working on something like this?

Additionally, the current approach of extracting a 'testtype' from the job is problematic for a number of reasons and there should be no a priori reason we couldn't just use the task label instead. We will have to deal with the change in the JobPriority table will mean the new task label keys will all be considered "new". jmaher stated that the new jobs will "age in" after 24 hours and we might be able to do this on a weekend when the push volume is down.

Additional thoughts on improvements are to introduce the capability to take jobs which are currently only run on mozilla-central and add them to autoland with a reduced schedule where they would run only every N pushes. Then we could change the mozilla-central jobs to run on a nightly scheduler instead on per push.

What else would we like?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

for the last idea (call it tier-2 on autoland), I am thinking every 25th or 30th push. It was specifically targeted at Perf jobs that we want to reduce the frequency but we still care about finding regressions. In the context of the original comment, this seems like a good idea for many other tests, not just perf tests- maybe everything tier-2 ?

Comment 2

[Tom Prince [:tomprince]]

(In reply to Bob Clary [:bc:] from comment #0)

Additionally, the current approach of extracting a 'testtype' from the job is problematic for a number of reasons and there should be no a priori reason we couldn't just use the task label instead. We will have to deal with the change in the JobPriority table will mean the new task label keys will all be considered "new". jmaher stated that the new jobs will "age in" after 24 hours and we might be able to do this on a weekend when the push volume is down.

I do think using the labels will work (and depending on the details of how SETA does its calculation, may not even need to age in; it can look at the treeherder data for older jobs which will have the label).

If, for some reason, the testtype is required or useful, though, I think it would be better to pass that data from taskgraph to SETA, rather than trying to extract it from the label.

Comment 3

[Bob Clary [:bc] (inactive)]

Armen: One thing that would help in developing seta would be the ability to test changes and see how things on production would actually be affected. I don't know how to do that but perhaps some of the work you mentioned in this week's meeting would help.

Comment 4

[Armen [:armenzg]]

With regards to this bug, I would suggest not spending too much energy improving SETA too much since we're hoping to move away from it.

My overall architectural requests is that determining what needs to be scheduled be completely removed from Treeherder. Even the day that I was finishing its integration into Treeherder I remember realizing that it was not the right place to live (much better though than where it used to be). The difficulty of testing SETA is because there are multiple systems generating data for each other.

FYI the testtype logic came from the Buildbot era when the builder name was all there was.

(In reply to Bob Clary [:bc:] from comment #3)

Armen: One thing that would help in developing seta would be the ability to test changes and see how things on production would actually be affected. I don't know how to do that but perhaps some of the work you mentioned in this week's meeting would help.

The only way I can think of testing this is by having an integration script that does:

1. On an m-c checkout execute the gecko decision code locally (assuming you have patches in your checkout)
2. Set up Treeherder for you (git checkout & docker-compose up; This will have the SETA APIs ready for you)
3. Have a way for Treeherder to consume the artifacts generated on step 1
4. Make the Gecko decision code point to http://localhost:8000 for the data it needs

A slight variation of the above would be to make such script to push code to try that will generate the modified artifacts, make the local checkout fetch the artifacts from the associated try push and execute locally the Gecko decision code that would use your local TH instance.

Comment 5

[Bob Clary [:bc] (inactive)]

seta is no more.