Load test symbols uploads



a year ago
a year ago


(Reporter: peterbe, Assigned: peterbe)


Firefox Tracking Flags

(Not tracked)



(2 attachments)



a year ago
On stage, the largest file I've managed to upload is 470MB. Anything larger than that always fails. According to the docker.app logs it's things like this::

 [2017-09-12 14:05:56 +0000] [9] [CRITICAL] WORKER TIMEOUT (pid:32)
 [2017-09-12 14:05:57 +0000] [40] [INFO] Booting worker with pid: 40

Also, the exception when that happens is like this:


That exception points to the fact that it basically fails to upload the zip file into the inbox. [0]

We have metrics of those uploads but the metrics never sends anything if the code errors out. 

First, try to get a handle on what our rate for uploading is.
Second, can we do anything with the Gunicorn timeouts to allow for longer requests. 

[0] https://github.com/mozilla-services/tecken/blob/364882998844dc138a1ec886098870bc0655bb30/tecken/upload/views.py#L218-L222

Comment 2

a year ago
The logging, on Dev is that it uploads the inbox file at 20.2MB/s. That means a 1GB file should take 50 seconds.

Comment 4

a year ago
Next time doing load testing by uploading by URL, see 

Comment 5

a year ago
Commit pushed to master at https://github.com/mozilla-services/tecken

bug 1399140 - longer gunicorn timeout (#393)

* bug 1399140 - longer gunicorn timeout

* default timeout


a year ago
Assignee: nobody → peterbe

Comment 8

a year ago
Miles, see https://github.com/mozilla-services/cloudops-deployment/pull/1140
It's going to allow me to experiment with the new 120s Gunicorn default timeout on Dev.

Comment 9

a year ago
Load testing on Stage is complete. I've been bombarding it with uploads, some in the 1GB range. And it just works. Repeatedly. No more timeouts. 

A new feature was added today [0] which will take some time to start benefitting from but that'll (according to local testing) significantly improve upload speeds since it reduces the amount of gzipping necessary. That new feature deserves to be kept an eye on but it'll just make things better. 

As far as load testing, Stage can handle it. 

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1410155
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.