Closed Bug 676086 Opened 8 years ago Closed 4 years ago

Detect performance regressions as failures in the endurance tests

Categories

(Mozilla QA Graveyard :: Infrastructure, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: davehunt, Unassigned)

References

Details

(Whiteboard: [blocked by release of tbpl 2])

We somehow need to identify a performance regression in the endurance tests as a failure and notify the appropriate people.
I don't think that this is something we can do in our tests. We could use the Jenkins history or would have to update the dashboard code to ensure such kind of regressions can be found. Please file an issue over there if it would still be important to do.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INVALID
I'd prefer to keep this one open, however I agree that this is probably the wrong component. I'm also not sure that the dashboard is the correct component though. I suspect we might need to look into reporting to DataZilla in order to identify regressions. Perhaps Mozmill should support reporting to DataZilla, or at least the automation scripts? Changing component to Mozmill Automation.
Status: RESOLVED → REOPENED
Component: Mozmill Tests → Mozmill Automation
Resolution: INVALID → ---
(In reply to Dave Hunt (:davehunt) from comment #2)
> component though. I suspect we might need to look into reporting to
> DataZilla in order to identify regressions. Perhaps Mozmill should support
> reporting to DataZilla, or at least the automation scripts? Changing
> component to Mozmill Automation.

I haven't touched datazilla yet nor any other perf related dashboard. Clint, what would you suggest? Do we already have a framework like talos which can auto-detect that, or with a bit of configuration?
Flags: needinfo?(ctalbert)
I think the new UI for datazilla should work, or we could create a custom UI similar to the b2g project. We should just need to massage the data into a format suitable for datazilla. That said, I wouldn't consider this a high priority at this time.
Yep, I agree with Dave. The way it would work is that you'd start another database on the datazilla system for mozmill and you would write your endurance data into it. Then datazilla can use its statistical modeling to determine if your current reporting is actually an error given the parameters you set (i.e. > 10% change is an error etc).

I also agree that this isn't important right now. Let's focus on TBPL v2 first so that mozmill correctness tests become *visible*. Then let's focus on wiring up the endurance tests to datazilla.
Flags: needinfo?(ctalbert)
Component: Mozmill Automation → Infrastructure
Status: REOPENED → NEW
Whiteboard: [blocked by release of tbpl 2]
We are no longer actively running or maintaining the endurance tests.
Status: NEW → RESOLVED
Closed: 7 years ago4 years ago
QA Contact: hskupin
Resolution: --- → WONTFIX
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.