Load test symbols uploads

RESOLVED FIXED

Status

Socorro
Symbols
RESOLVED FIXED
3 months ago
2 months ago

People

(Reporter: peterbe, Assigned: peterbe)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Assignee)

Description

3 months ago
On stage, the largest file I've managed to upload is 470MB. Anything larger than that always fails. According to the docker.app logs it's things like this::

 [2017-09-12 14:05:56 +0000] [9] [CRITICAL] WORKER TIMEOUT (pid:32)
 [2017-09-12 14:05:57 +0000] [40] [INFO] Booting worker with pid: 40

Also, the exception when that happens is like this:

 https://sentry.prod.mozaws.net/operations/symbols-stage/issues/647537/

That exception points to the fact that it basically fails to upload the zip file into the inbox. [0]

We have metrics of those uploads but the metrics never sends anything if the code errors out. 

First, try to get a handle on what our rate for uploading is.
Second, can we do anything with the Gunicorn timeouts to allow for longer requests. 


[0] https://github.com/mozilla-services/tecken/blob/364882998844dc138a1ec886098870bc0655bb30/tecken/upload/views.py#L218-L222

Comment 1

3 months ago
Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/c4c6d49926e0aa11eba26fd01838e0f3058bb071
bug 1399140 - more verbose about symbol uploads
(Assignee)

Comment 2

3 months ago
The logging, on Dev is that it uploads the inbox file at 20.2MB/s. That means a 1GB file should take 50 seconds.
Created attachment 8907132 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/393
(Assignee)

Comment 4

3 months ago
Next time doing load testing by uploading by URL, see 
https://bugzilla.mozilla.org/show_bug.cgi?id=1392400#c15

Comment 5

3 months ago
Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/fe5f488c1b13d563f597872ade4f6b6f3a35b2e2
bug 1399140 - longer gunicorn timeout (#393)

* bug 1399140 - longer gunicorn timeout

* default timeout
Created attachment 8908610 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/400

Comment 7

3 months ago
Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/8bd814b2e705e8b0544a89164cb374f345230b3e
bug 1399140 - much longer default Gunicorn timeout (#400)
(Assignee)

Updated

3 months ago
Assignee: nobody → peterbe
(Assignee)

Comment 8

3 months ago
Miles, see https://github.com/mozilla-services/cloudops-deployment/pull/1140
It's going to allow me to experiment with the new 120s Gunicorn default timeout on Dev.
(Assignee)

Comment 9

2 months ago
Load testing on Stage is complete. I've been bombarding it with uploads, some in the 1GB range. And it just works. Repeatedly. No more timeouts. 

A new feature was added today [0] which will take some time to start benefitting from but that'll (according to local testing) significantly improve upload speeds since it reduces the amount of gzipping necessary. That new feature deserves to be kept an eye on but it'll just make things better. 

As far as load testing, Stage can handle it. 


[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1410155
Status: NEW → RESOLVED
Last Resolved: 2 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.