Closed Bug 457780 Opened 11 years ago Closed 6 years ago

Write tests for regression tracking per-machine


(Webtools Graveyard :: Graph Server, defect)

Not set


(Not tracked)



(Reporter: rdoherty, Assigned: lars)


Sometimes a machine will report back test results that are way off from the other 2. We should have tests that run periodically and verify the test results are no more than a certain % difference.
Assignee: nobody → mark
Assignee: mark → lars
before any serious work can be done on this, there are some questions that need to be answered:

1 - how does this application run?  cron?  invoked by a Web app?  
2 - how does this app get configured?  how does one specify the "certain % difference" ?
3 - in what form does this app report its findings?  return data via stdout? generates a web page?  generates json or xml data?  updates stats in a database?
Initial specs:

1) Daily via cron - benchmarking needed to determine if we can decrease this interval
2) Test for more than 1 standard deviation from the previous 3 days of data from the latest test run
    Flag machine for problems, manually verify fixed or 
    Also need to flag tests

3) HTML page that shows all machines as red or green
    List failed tests under machines

We will need some way of un-flagging a machine or test one the issue has been resolved. Command line tool might be easiest, avoids the need for user accounts, etc.

We will need some way to track what machines and tests were flagged as failed. A table or two in the database will most likely be necessary.
Summary: Write tests for machine agreement → Write tests for regression tracking per-machine
old, outdated
Closed: 6 years ago
Resolution: --- → WONTFIX
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.