Closed Bug 457780 Opened 11 years ago Closed 6 years ago
Write tests for regression tracking per-machine
Sometimes a machine will report back test results that are way off from the other 2. We should have tests that run periodically and verify the test results are no more than a certain % difference.
before any serious work can be done on this, there are some questions that need to be answered: 1 - how does this application run? cron? invoked by a Web app? 2 - how does this app get configured? how does one specify the "certain % difference" ? 3 - in what form does this app report its findings? return data via stdout? generates a web page? generates json or xml data? updates stats in a database?
Initial specs: 1) Daily via cron - benchmarking needed to determine if we can decrease this interval 2) Test for more than 1 standard deviation from the previous 3 days of data from the latest test run Flag machine for problems, manually verify fixed or Also need to flag tests 3) HTML page that shows all machines as red or green List failed tests under machines We will need some way of un-flagging a machine or test one the issue has been resolved. Command line tool might be easiest, avoids the need for user accounts, etc. We will need some way to track what machines and tests were flagged as failed. A table or two in the database will most likely be necessary.
Summary: Write tests for machine agreement → Write tests for regression tracking per-machine
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.