queries are getting locked on high load

RESOLVED FIXED

Status

--
blocker
RESOLVED FIXED
8 years ago
8 years ago

People

(Reporter: tarek, Assigned: tarek)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

8 years ago
On high load, SQL queries sent through pymysql + gevent to insert wbos into collections by batches get locked. 

We encounter this issue in the load test server and were able to fix it by lowering the size of the batches sent to MySql.

We don't know yet what's really happening, and if the driver is to blame, or if it's a problem on the database side, as the lock happens when the same table gets updated concurrently for the same user.

I am going to work today on the issue to try to it narrow down with a small reproducible test.

I have also prepared in bug 667731 everything needed to try an alternative backend on the load infra.
(Assignee)

Comment 1

8 years ago
Created attachment 542440 [details]
Sample to reproduce the lock

This WSGI server reproduces the issue. If you call it 10 times in a row with "curl http://localhost:8888" it will work on async mode.

If you decomment the gevent call, making it async, it will lock.

Now that I have a reproducible test I am going to try the alternative connector, and if it also fails, dig in the greenlets code
(Assignee)

Comment 2

8 years ago
I also have failures (no lock this time) with myconnpy. digging.

btw, select sleep(x) works well, so it might be related to the query in play
(Assignee)

Comment 3

8 years ago
Applying this patch in pymysql fixes the issue in my test !

http://groups.google.com/group/pymysql-users/browse_thread/thread/4c03e54d77e38f54#

We need to try this in the load test environment now, with batches of 100
(Assignee)

Updated

8 years ago
Blocks: 667731
(Assignee)

Comment 4

8 years ago
The previous patch did not resolve the lock in all cases, but this other patch found by Pete seem to work prefectly (applied on PyMySQL):

https://bugs.launchpad.net/myconnpy/+bug/711520

We've pushed a custom version on our internal PyPI repo
(Assignee)

Comment 5

8 years ago
The load test was a success. 

I'll follow up with the PyMySQL maintainer, and we're using our own patched version in the interim.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED

Comment 7

8 years ago
v2 patch at http://dl.dropbox.com/u/7533709/PyMySQL-0.4-sock-sendall-fix-v2.patch that fixes all four occurrences of sock.send()
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Blocks: 669526
latest patch in prod.
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.