676086 - Detect performance regressions as failures in the endurance tests

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We somehow need to identify a performance regression in the endurance tests as a failure and notify the appropriate people.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I don't think that this is something we can do in our tests. We could use the Jenkins history or would have to update the dashboard code to ensure such kind of regressions can be found. Please file an issue over there if it would still be important to do.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I'd prefer to keep this one open, however I agree that this is probably the wrong component. I'm also not sure that the dashboard is the correct component though. I suspect we might need to look into reporting to DataZilla in order to identify regressions. Perhaps Mozmill should support reporting to DataZilla, or at least the automation scripts? Changing component to Mozmill Automation.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Dave Hunt (:davehunt) from comment #2)
> component though. I suspect we might need to look into reporting to
> DataZilla in order to identify regressions. Perhaps Mozmill should support
> reporting to DataZilla, or at least the automation scripts? Changing
> component to Mozmill Automation.

I haven't touched datazilla yet nor any other perf related dashboard. Clint, what would you suggest? Do we already have a framework like talos which can auto-detect that, or with a bit of configuration?

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I think the new UI for datazilla should work, or we could create a custom UI similar to the b2g project. We should just need to massage the data into a format suitable for datazilla. That said, I wouldn't consider this a high priority at this time.

Comment 5

[cmtalbert]

Yep, I agree with Dave. The way it would work is that you'd start another database on the datazilla system for mozmill and you would write your endurance data into it. Then datazilla can use its statistical modeling to determine if your current reporting is actually an error given the parameters you set (i.e. > 10% change is an error etc).

I also agree that this isn't important right now. Let's focus on TBPL v2 first so that mozmill correctness tests become *visible*. Then let's focus on wiring up the endurance tests to datazilla.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We are no longer actively running or maintaining the endurance tests.