1011548 - Do not alert on try

Status: RESOLVED FIXED

(Datazilla Graveyard :: Metrics, defect)

Description

[Kyle Lahnakoski [:ekyle]]

Talos alerts automatically pickup all tests for all branches.  Add checks so we can disable alerting for specific branches, like Try.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I could see developers wanting alerts, but these would have to be instantaneous, so for our typical algorithms, alerting on try is not helpful

Comment 2

[Kyle Lahnakoski [:ekyle]]

That's good to know. 

Some thoughts: If I enable a ttest, we can alert sooner but with lower confidence, which may be good for developers who are only interested in their single changeset anyway.  The dzAlerts can already direct alert mails based on the type of test; maybe we can direct Try ttest alerts to another mail list.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

For try server, we want to compare against a master branch (mozilla-central, mozilla-inbound, fx-team, etc.).

Without knowing the branch we based our push off, we can't realistically compare.

Ideally we would do this:
* look at last patch, find the parent
* take parent and look for it in 24 hour history of pushlog for all branches
* compare current results as a next push from parent/branch - a ttest would be good- we just want to identify possible regressions

Of course, this could better be solved by compare.py or compare-talos, ideally this would be part of a bigger collection of tools that we can do a better job of utilizing try server to reduce our overall regressions introduced.

Another way to look at this is:
* look at the last patch in the queue, find the parent
* look for the parent in the data results table (it should always be the correct branch)
* temporarily insert try revision into job history and do a ttest to look for a change.

How do you solve:
* missing data? (I believe here we need to walk the pushlog until we find data)
* retriggers? (ignore previous data set and treat this as a new data set)

Comment 4

[Kyle Lahnakoski [:ekyle]]

Try is no longer analyzed