976102 - Load test the APK generator/signer services

Status: RESOLVED FIXED

(Marketplace Graveyard :: Integration, defect, P1)

Description

[Kumar McMillan [:kumar]]

In preparation to keep moving the APK generator/signer services through the Fennec release channels, let's do some load testing that will simulate some realistic traffic patterns that we anticipate.

Comment 1

[Austin King [:ozten]]

I load tested the prototype. Those scripts may or may not be useful.

I have not load tested since we added the APK Signer and S3.

Comment 2

[Austin King [:ozten]]

Do you guys have a format or any guidelines?

I'm load testing this and have found 4 or 5 bottlenecks which I've fixed.

I guess I'm wondering how to report findings and what are useful data points.

Comment 3

[Austin King [:ozten]]

Attachment: combined.png

ab test output, combined data from 9 runs with
* 1 concurrent user for 15 minutes
* 2 concurrent user for 15 minutes
* 4 concurrent user for 15 minutes
* etc 8, 16, 32, 64, 128
* up to 256 concurrent user for 15 minutes

Comment 4

[Austin King [:ozten]]

Attachment: app1.png

Additonally 1 concurrent user for /app_updates was made for 135 minutes.

Comment 5

[Austin King [:ozten]]

Attachment: load64.png

Data for ab load for a 3MB apk with 64 concurrent users for 15 minutes.

Concurrency Level:      64
Time taken for tests:   900.074 seconds
Complete requests:      3139
Failed requests:        4
   (Connect: 0, Receive: 0, Length: 2, Exceptions: 2)
Write errors:           0
Total transferred:      10069289415 bytes
HTML transferred:       10068688191 bytes
Requests per second:    3.49 [#/sec] (mean)
Time per request:       18351.312 [ms] (mean)
Time per request:       286.739 [ms] (mean, across all concurrent requests)
Transfer rate:          10924.97 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  498 1616.3    403   63493
Processing:  4419 17684 6448.5  17874  127248
Waiting:      140  258 247.6    198    9104
Total:       4849 18182 6653.1  18304  127248

Percentage of the requests served within a certain time (ms)
  50%  18301
  66%  20875
  75%  22527
  80%  23439
  90%  25660
  95%  27568
  98%  29370
  99%  30811
 100%  127248 (longest request)

Comment 6

[Austin King [:ozten]]

Stage was load tested with 1 Controller (bug#976170) and 2 Generators.

64 concurrent users seems starts to be the breaking point for exceptions, although these were exceptions at the client end, not the server end. Requests took 18 seconds on average instead of less than 3 seconds which is what we were seeing for a 3MB apk with only 1 concurrent user.

Assuming 14 hours of active traffic per day and a 5 second session, our current setup should be able to support 600,000 daily users.

This should be good for Android release channel. It is a high priority to fix 976170, to allow for scaling to 4 million daily users.