1408592 - Avoid running out of RDS connections

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[Peter Bengtsson [:peterbe]]

With RDS the amount of connections you can create is limited [0]. 

See https://sentry.prod.mozaws.net/operations/symbols-stage/issues/666658/

We can either with to persistent connections [1] or we can try to send a connection into each future when using a ThreadPoolExecutor pool.


[0] https://stackoverflow.com/a/38555806/205832
[1] https://devcenter.heroku.com/articles/python-concurrency-and-database-connections#persistent-connections and https://docs.djangoproject.com/en/1.11/ref/databases/#persistent-connections

Comment 1

[Peter Bengtsson [:peterbe]]

I can't find any way to supply a custom connection or cursor to the ORM. Probably best to use persistent connections.

Comment 2

[Peter Bengtsson [:peterbe]]

Another option worth considering is to not do any saves inside the upload_file_upload() function. Instead it could be made to return FileUpload *instances* and then in the archive_upload() view call FileUpload.bulk_create(file_uploads).

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/477

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/1745f53f79eb634a6fc71ae6707a1562acad375d
fixes bug 1408592 - setting CONN_MAX_AGE for persistent connections (#477)