1206228 - [tools] services-qa-dashboard

Status: RESOLVED INVALID

(Cloud Services :: QA: Test Automation, defect)

Description

[Richard Pappalardo [:rpapa][:rpappalardo]]

Summary

* Create a (top-level) dashboard of KPIs for Cloud Services QA team
* example: click on Test Coverage KPI Widget, and drilldown to test type widgets
* see: https://etherpad.mozilla.org/services-qa-dashboard

Defects

* defects-blockers (blocking current release - from stage to pre-prod )
* p1, p2 (blockers)
* would be in Depends field of deployment bug
* defects found in prod

Strategy: 

* all p1,p2s to deployment tickets in depends/blocks field


Test Coverage

* e2e coverage for cloud services projects (by test type) into Bugzilla
* bz filter to count those that are in verified state
* goal 100% coverage
** example: an application w/ all test-types covered (e2e-test, schema-check, etc.) = 100% coverage


Deploy Cycle KPI Widgets (Prod & Stage)

* dashboard - 
** scatterplot graph with a median (max, min)
** i.e. http://www.scielo.br/img/revistas/spmj/v122n6/a05fig03.gif
** appropriate drilldown to a dashboard with similar graphs for each project

500s

* # of users  experiencing issues w/ your product at any given time
* example:
** x/1000 users have a problem
** non 200 or 300 will look to the user like an issue
** # of 400s, 500s out of 1000000 total requests
* hour snapshot post to datadog?

# of projects X * (elapsed uptime hours - total downtime hours)
- e.g 10000 hours uptime - 2 hours downtime = 2/10000