Load test symbols uploads

RESOLVED FIXED

Status

RESOLVED FIXED
a year ago
a year ago

People

(Reporter: peterbe, Assigned: peterbe)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Assignee)

Description

a year ago
On stage, the largest file I've managed to upload is 470MB. Anything larger than that always fails. According to the docker.app logs it's things like this::

 [2017-09-12 14:05:56 +0000] [9] [CRITICAL] WORKER TIMEOUT (pid:32)
 [2017-09-12 14:05:57 +0000] [40] [INFO] Booting worker with pid: 40

Also, the exception when that happens is like this:

 https://sentry.prod.mozaws.net/operations/symbols-stage/issues/647537/

That exception points to the fact that it basically fails to upload the zip file into the inbox. [0]

We have metrics of those uploads but the metrics never sends anything if the code errors out. 

First, try to get a handle on what our rate for uploading is.
Second, can we do anything with the Gunicorn timeouts to allow for longer requests. 


[0] https://github.com/mozilla-services/tecken/blob/364882998844dc138a1ec886098870bc0655bb30/tecken/upload/views.py#L218-L222
(Assignee)

Comment 2

a year ago
The logging, on Dev is that it uploads the inbox file at 20.2MB/s. That means a 1GB file should take 50 seconds.
(Assignee)

Comment 4

a year ago
Next time doing load testing by uploading by URL, see 
https://bugzilla.mozilla.org/show_bug.cgi?id=1392400#c15

Comment 5

a year ago
Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/fe5f488c1b13d563f597872ade4f6b6f3a35b2e2
bug 1399140 - longer gunicorn timeout (#393)

* bug 1399140 - longer gunicorn timeout

* default timeout
(Assignee)

Updated

a year ago
Assignee: nobody → peterbe
(Assignee)

Comment 8

a year ago
Miles, see https://github.com/mozilla-services/cloudops-deployment/pull/1140
It's going to allow me to experiment with the new 120s Gunicorn default timeout on Dev.
(Assignee)

Comment 9

a year ago
Load testing on Stage is complete. I've been bombarding it with uploads, some in the 1GB range. And it just works. Repeatedly. No more timeouts. 

A new feature was added today [0] which will take some time to start benefitting from but that'll (according to local testing) significantly improve upload speeds since it reduces the amount of gzipping necessary. That new feature deserves to be kept an eye on but it'll just make things better. 

As far as load testing, Stage can handle it. 


[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1410155
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.