1208292 - permanent location for SETA data that supports both taskcluster and buildbot

Status: RESOLVED DUPLICATE of bug 1243123

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

SETA data is used to coalesce tests on select branches

The data is parsed from this server
http://alertmanager.allizom.org/data/setadetails
and our buildbot BRANCH dictionaries are updated to reflect the tests than can be skipped.

In bug 1208186, we discuss the need to move this to a production server because the current one is just for development.

I'm working on bug 1176784 to store these configs the buildbot configs so reconfigs are not dependent on the server with this data if it should be unavailable.  

However, given that this functionality will also be needed in taskcluster, perhaps it should be in a location that is accessible to both systems.  Joel suggested today that these configs be stored in tree and updated from the database as needed.  

Thoughts?

Comment 1

[Kim Moir [:kmoir] ET]

Selena, who would you suggest I talk to on the taskcluster side of things regarding coalescing requirements?

Comment 2

[Selena Deckelmann :selenamarie :selena]

Let's talk to :jonasfj when he's back from PTO next week.

I like Joel's line of thinking! :)

Comment 3

[Kim Moir [:kmoir] ET]

Sounds good, thanks Selena

Comment 4

[Kim Moir [:kmoir] ET]

:jonasfj what are your thoughts on this?  Would it be easier to setup a meeting to discuss?

Comment 5

[Jonas Finnemann Jensen (:jonasfj)]

I'm not entirely sure what these configs are... or what they are used for. I got the impression that:
 A) SETA was about scheduling some tasks less frequently based on statistics, and
 B) coalescing was about merging tasks after they are scheduled, if the queue of pending tasks is long
I guess both might be viewed by the end-user as coalescence. But if (B) doesn't need stats from SETA,
then perhaps we can use terms:
  *static-coalescence* to mean (A), and
  *dynamic-coalescence* to mean (B)
Better terms are welcome, I just need to know what we're talking about.

----
Assuming this is correct, stats/decisions/data from SETA is probably something we need to inject into
the decision task that schedules tasks initially (ie. the place we decide to statically 
coalesce some tasks).

And then stats/decisions/data from SETA doesn't seem to be needed inside the coalescence service that
monitors number of pending tasks and collaborates with workers on dynamic coalescence when there is a
lot of pending tasks. 

> Joel suggested today that these configs be stored in tree and updated from the database as needed.
If the above is correct, and these configs/stats/decisions/data can fit elegantly into the tree,
this seems like a very good solution, as nothing will be affected if SETA crashes.
If the failure case of SETA is that we rely on old stats, that seems very robust! :)
(I won't imagine we could rely on old stats for days, if not weeks)

I generally (sometimes irrationally) fear all forms of mutable storage that isn't managed,
highly available, under SLA and offering tangible scalability targets. So not introducing a new one
would be great. If decision tasks called out to SETA for stats/data/decisions/configs, that would be
a new thing.

---
Unrelated: not sure how you get your data, but we do have pulse exchanges with everything. But if you
get from treeherder you should be fine.

Comment 6

[Kim Moir [:kmoir] ET]

Thanks Jonas for the information.  Joel, where and in what format do you propose storing these configs in tree?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

I am thinking something like our runtimes which are updated manually as needed:
https://dxr.mozilla.org/mozilla-central/source/testing/runtimes

What we would do here is add a .json file which would match the output of the api.  The problem is that we have branch specifics, maybe we could create .json file with either branch names like mozilla-inbound.json, or just have the json be:
{'mozilla-inbound': jsondata,
 'fx-team': jsondata}

Give me some general direction and we can make it work.  I imagine from the buildbot side we could pull from in-tree/hg, I know mozharness could easily do that, not sure if buildbot proper can easily do that.

Comment 8

[Armen [:armenzg]]

Should we dupe this to bug 1243123?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

I think we should dup this bug, we have instances that work with api endpoints for both taskcluster and buildbot, so I think this is a duplicate :)