1236973 - [loadtest] Create loadtest summary report template

Status: RESOLVED INVALID

(Cloud Services :: QA: Test Automation, defect)

Description

[Richard Pappalardo [:rpapa][:rpappalardo]]

To provide consistent test reporting for all services, we need a template for reporting a summary of loadtest results.  Should summary results be provided in STAGE deployment tickets as final verification criteria?

* DataDog will be 'source of truth' w/ loadsv2, but is in graphical format and does not provide a "summary" format we could use 
* DataDog has a good API for CLI queries (per bobm) - need to investigate
* beeswithmachineguns has a useful output to console that we might use as starting point for data points.

Having a clear summary of test results will also be useful for making deployment 'go / no go' decisions.  Determining thresholds will be defined here: Bug 1236970.

Comment 1

[Richard Pappalardo [:rpapa][:rpappalardo]]

EXAMPLES:

Summaries of bee runs:
https://docs.google.com/spreadsheets/d/1RInD6equXLvhyZvsjVxieqz_LSoYU5OGdIus7AommQc/edit#gid=771043189


Sample output from bees run:

./bees attack --use-siege -c 16 -w 300s -u <post URL here>

Concurrency Level:      15
Complete requests:      108374
Failed requests:        0
Non-2xx responses:      0
Total Transferred:      5680964567 bytes
Requests per second:    362.25 [#/sec] (mean)
Time per request:       40.000 [ms] (mean)
50% response time:      40 [ms] (mean)
75% response time:      50 [ms] (mean)
90% response time:      60 [ms] (mean)
95% response time:      60 [ms] (mean)
99% response time:      70 [ms] (mean)

Comment 2

[Richard Pappalardo [:rpapa][:rpappalardo]]

We'll track this in googledocs going forward.