Docker set up - OperationalError: Too many connections
Categories
(Tree Management :: Treeherder: Infrastructure, defect)
Tracking
(Not tracked)
People
(Reporter: camd, Assigned: armenzg)
Details
Attachments
(2 files)
Aryx and I got this while running the full-stack with ingestion in Docker.
I had let it run for maybe 15 minutes before getting this error:
backend | [2020-04-15 15:30:48,120] ERROR [django.request:228] Internal Server Error: /api/failureclassification/
backend | Traceback (most recent call last):
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection
backend | self.connect()
backend | File "/usr/local/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner
backend | return func(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 197, in connect
backend | self.connection = self.get_new_connection(conn_params)
backend | File "/usr/local/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner
backend | return func(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 233, in get_new_connection
backend | return Database.connect(**conn_params)
backend | File "/usr/local/lib/python3.7/site-packages/MySQLdb/__init__.py", line 84, in Connect
backend | return Connection(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/MySQLdb/connections.py", line 179, in __init__
backend | super(Connection, self).__init__(*args, **kwargs2)
backend | MySQLdb._exceptions.OperationalError: (1040, 'Too many connections')
backend |
backend | The above exception was the direct cause of the following exception:
backend |
backend | Traceback (most recent call last):
backend | File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
backend | response = get_response(request)
backend | File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
backend | response = self.process_exception_by_middleware(e, request)
backend | File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
backend | response = wrapped_callback(request, *callback_args, **callback_kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
backend | return view_func(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/viewsets.py", line 114, in view
backend | return self.dispatch(request, *args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 505, in dispatch
backend | response = self.handle_exception(exc)
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 465, in handle_exception
backend | self.raise_uncaught_exception(exc)
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
backend | raise exc
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 502, in dispatch
backend | response = handler(request, *args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/mixins.py", line 46, in list
backend | return Response(serializer.data)
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 760, in data
backend | ret = super().data
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 260, in data
backend | self._data = self.to_representation(self.instance)
backend | File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 678, in to_representation
backend | self.child.to_representation(item) for item in iterable
backend | File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 276, in __iter__
backend | self._fetch_all()
backend | File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1261, in _fetch_all
backend | self._result_cache = list(self._iterable_class(self))
backend | File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 57, in __iter__
backend | results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
backend | File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1149, in execute_sql
backend | cursor = self.connection.cursor()
backend | File "/usr/local/lib/python3.7/site-packages/debug_toolbar/panels/sql/tracking.py", line 54, in cursor
backend | connection._djdt_cursor(*args, **kwargs), connection, panel
backend | File "/usr/local/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner
backend | return func(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 260, in cursor
backend | return self._cursor()
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 236, in _cursor
backend | self.ensure_connection()
backend | File "/usr/local/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner
backend | return func(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection
backend | self.connect()
backend | File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
backend | raise dj_exc_value.with_traceback(traceback) from exc_value
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection
backend | self.connect()
backend | File "/usr/local/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner
backend | return func(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 197, in connect
backend | self.connection = self.get_new_connection(conn_params)
backend | File "/usr/local/lib/python3.7/site-packages/django/utils/asyncio.py", line 26, in inner
backend | return func(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/django/db/backends/mysql/base.py", line 233, in get_new_connection
backend | return Database.connect(**conn_params)
backend | File "/usr/local/lib/python3.7/site-packages/MySQLdb/__init__.py", line 84, in Connect
backend | return Connection(*args, **kwargs)
backend | File "/usr/local/lib/python3.7/site-packages/MySQLdb/connections.py", line 179, in __init__
backend | super(Connection, self).__init__(*args, **kwargs2)
backend | django.db.utils.OperationalError: (1040, 'Too many connections')
|
Reporter | |
Comment 1•4 years ago
|
||
Armen: Could this be related to the new data-ingestion work that was merged?
|
Assignee | |
Comment 2•4 years ago
|
||
When does the API api/failureclassification get hit?
Were you trying to do use the UI? (I'm trying to figure out if that's how the API got hit)
Were you using custom DATABASE_URL? There was a spike of connections on the replica on the 15th.
Were you just running docker-compose up?
Did you have any local code changes?
For the record, the recent changes only ingest pushes and tasks.
|
||
Comment 3•4 years ago
|
||
Saw this a few times today, just reloading the view I had open (~40 pushes on autoland) was enough to trigger the MySQL connection issue.
The read-only replica is used as DATABASE_URL, and only docker-compose up interacts with the container content.
|
|
Assignee | |
Comment 4•4 years ago
|
||
The local mysql container has the default 151 connections.
Production has a maximum of 2598 and the replica has a maximum of 1282.
After speaking with Aryx he was not using the replice.
I will double the number of connections for now.
|
|
Assignee | |
Updated•4 years ago
|
|
||
Comment 5•4 years ago
|
||
|
|
Assignee | |
Updated•4 years ago
|
|
|
Reporter | |
Comment 6•4 years ago
|
||
I still get this after maybe 20 minutes of ingesting locally in Docker.
I followed the steps in the docs to do ingestion:
docker-compose down
docker-compose run frontend sh -c "yarn && yarn build"
docker-compose up --build
docker-compose run backend celery -A treeherder worker -B --concurrency 5
|
|
Assignee | |
Comment 7•4 years ago
|
||
(In reply to Cameron Dawson [:camd] from comment #6)
I still get this after maybe 20 minutes of ingesting locally in Docker.
I followed the steps in the docs to do ingestion:
docker-compose down docker-compose run frontend sh -c "yarn && yarn build" docker-compose up --build docker-compose run backend celery -A treeherder worker -B --concurrency 5
If you have a lower concurrency you will not be hitting this.
Also, were you trying to ingest everything besides pushes & tasks?
By default, it ingests from these queues store_pulse_pushes,store_pulse_tasks.
Here are some of the other options if you want to narrow down the ingestion:
log_parser,log_parser_fail,log_autoclassify,log_autoclassify_fail,default,generate_perf_alerts,pushlog,seta_analyze_failures
|
||
Comment 8•4 years ago
|
||
I hit this issue with the default settings and this command that's in the docs: docker-compose run backend celery -A treeherder worker -B --concurrency 5. I have not tried adjusting the concurrency. What should it be adjusted to? If that's the issue, then the docs should be updated.
FWIW, I don't recall it doing this in the past so I'm curious what has changed in the past 6 months.
|
|
||
Comment 9•4 years ago
|
||
|
|
Assignee | |
Comment 10•4 years ago
|
||
(In reply to Sarah Clements [:sclements] from comment #8)
I hit this issue with the default settings and this command that's in the docs:
docker-compose run backend celery -A treeherder worker -B --concurrency 5. I have not tried adjusting the concurrency. What should it be adjusted to? If that's the issue, then the docs should be updated.FWIW, I don't recall it doing this in the past so I'm curious what has changed in the past 6 months.
Where you just running docker-compose up?
|
|
||
Comment 11•4 years ago
•
|
||
I think I mostly run docker-compose up --build. Is there a difference in connections between the two? Although, admittedly I don't always run docker-compose run frontend sh -c "yarn && yarn build" first.
|
|
Assignee | |
Comment 12•4 years ago
|
||
(In reply to Sarah Clements [:sclements] from comment #11)
I think I mostly run
docker-compose up --build. Is there a difference in connections between the two?
The two are similar. One will check to see if there are new requirements that need installing.
Although, admittedly I don't always run
docker-compose run frontend sh -c "yarn && yarn build"first.
That's only if you want to build the frontend in a similar manner as Heroku, however, you loose hot reloading.
Alternatively, you don't need to run that command with Docker if you have Node installed.
It's not needed unless you're debugging some odd webpack/Neutrino issues.
|
|
||
Comment 13•4 years ago
|
||
hrm, I think the reason I started doing that was because switching between branches in the same container wouldn't update packages correctly. Should I be killing the container and initiating it again instead? And then just using docker-compose up?
|
|
Assignee | |
Comment 14•4 years ago
|
||
Using docker-compose up --build is the best option. No need to call docker-compose down.
|
|
Assignee | |
Comment 15•4 years ago
|
||
The changes landed.
I expect this to be better now. Thanks for the review!
Description
•