1399140 - Load test symbols uploads

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[Peter Bengtsson [:peterbe]]

On stage, the largest file I've managed to upload is 470MB. Anything larger than that always fails. According to the docker.app logs it's things like this::

 [2017-09-12 14:05:56 +0000] [9] [CRITICAL] WORKER TIMEOUT (pid:32)
 [2017-09-12 14:05:57 +0000] [40] [INFO] Booting worker with pid: 40

Also, the exception when that happens is like this:

 https://sentry.prod.mozaws.net/operations/symbols-stage/issues/647537/

That exception points to the fact that it basically fails to upload the zip file into the inbox. [0]

We have metrics of those uploads but the metrics never sends anything if the code errors out. 

First, try to get a handle on what our rate for uploading is.
Second, can we do anything with the Gunicorn timeouts to allow for longer requests. 


[0] https://github.com/mozilla-services/tecken/blob/364882998844dc138a1ec886098870bc0655bb30/tecken/upload/views.py#L218-L222

Comment 1

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/c4c6d49926e0aa11eba26fd01838e0f3058bb071
bug 1399140 - more verbose about symbol uploads

Comment 2

[Peter Bengtsson [:peterbe]]

The logging, on Dev is that it uploads the inbox file at 20.2MB/s. That means a 1GB file should take 50 seconds.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/393

Comment 4

[Peter Bengtsson [:peterbe]]

Next time doing load testing by uploading by URL, see 
https://bugzilla.mozilla.org/show_bug.cgi?id=1392400#c15

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/fe5f488c1b13d563f597872ade4f6b6f3a35b2e2
bug 1399140 - longer gunicorn timeout (#393)

* bug 1399140 - longer gunicorn timeout

* default timeout

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/400

Comment 7

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/8bd814b2e705e8b0544a89164cb374f345230b3e
bug 1399140 - much longer default Gunicorn timeout (#400)

Comment 8

[Peter Bengtsson [:peterbe]]

Miles, see https://github.com/mozilla-services/cloudops-deployment/pull/1140
It's going to allow me to experiment with the new 120s Gunicorn default timeout on Dev.

Comment 9

[Peter Bengtsson [:peterbe]]

Load testing on Stage is complete. I've been bombarding it with uploads, some in the 1GB range. And it just works. Repeatedly. No more timeouts. 

A new feature was added today [0] which will take some time to start benefitting from but that'll (according to local testing) significantly improve upload speeds since it reduces the amount of gzipping necessary. That new feature deserves to be kept an eye on but it'll just make things better. 

As far as load testing, Stage can handle it. 


[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1410155