1228154 - Generate new performance alerts as data is ingested

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect)

Description

[William Lachance (:wlach)]

We should generate new performance alerts as data is ingested.

I think ideally we would do this whenever a performance datum object is saved, however since we have no fast way of finding out whether a datum's series is a subtest (we don't want to generate alerts for subtests) such an operation would be slow. Bug 1226693 should address this, by making it possible to get this information from the database (rather than needing to parse out each series' extra_properties column one-by-one, which takes about a second on treeherder-stage, in my testing).

For now, let's generate them as performance data is ingested, in the ETL layer (https://github.com/mozilla/treeherder/blob/master/treeherder/etl/perf.py). This is easy to implement and we can move to a more generic and easier to test implementation once it's available (hopefully soon).

Comment 1

[William Lachance (:wlach)]

Attachment: PR

Hey Ed, if you have a moment tomorrow, could you take a look and let me know if you see any red flags in this? It's meant to (maybe) generate performance alerts as new data comes in (basically whenever a talos job completes).

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8692667 [details] [review]
PR

Mauro, could you do a final review on this? I've fixed up the issues Ed mentioned and added unit tests. Things are now in testing on try and seem to be working well.

Comment 3

[Mauro Doglio [:mdoglio]]

Comment on attachment 8692667 [details] [review]
PR

I left a few minor comments, fix it and ship it!

Comment 4

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/45caf5de80bd075fe61a86514170b0479ce1444b
Bug 1228154 - Use fixtures for performance ingestion unit tests

It's easier to reuse code and reason about it this way

https://github.com/mozilla/treeherder/commit/a8e663d61d912c9f9f00fca2de386f9fb7929590
Bug 1228154 - Generate new performance alerts as data is ingested