Closed Bug 667846 Opened 12 years ago Closed 12 years ago

queries are getting locked on high load

Categories

(Cloud Services :: Server: Sync, defect)

x86_64
Linux
defect
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: tarek, Assigned: tarek)

References

Details

Attachments

(1 file)

On high load, SQL queries sent through pymysql + gevent to insert wbos into collections by batches get locked. 

We encounter this issue in the load test server and were able to fix it by lowering the size of the batches sent to MySql.

We don't know yet what's really happening, and if the driver is to blame, or if it's a problem on the database side, as the lock happens when the same table gets updated concurrently for the same user.

I am going to work today on the issue to try to it narrow down with a small reproducible test.

I have also prepared in bug 667731 everything needed to try an alternative backend on the load infra.
This WSGI server reproduces the issue. If you call it 10 times in a row with "curl http://localhost:8888" it will work on async mode.

If you decomment the gevent call, making it async, it will lock.

Now that I have a reproducible test I am going to try the alternative connector, and if it also fails, dig in the greenlets code
I also have failures (no lock this time) with myconnpy. digging.

btw, select sleep(x) works well, so it might be related to the query in play
Applying this patch in pymysql fixes the issue in my test !

http://groups.google.com/group/pymysql-users/browse_thread/thread/4c03e54d77e38f54#

We need to try this in the load test environment now, with batches of 100
Blocks: 667731
The previous patch did not resolve the lock in all cases, but this other patch found by Pete seem to work prefectly (applied on PyMySQL):

https://bugs.launchpad.net/myconnpy/+bug/711520

We've pushed a custom version on our internal PyPI repo
The load test was a success. 

I'll follow up with the PyMySQL maintainer, and we're using our own patched version in the interim.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
v2 patch at http://dl.dropbox.com/u/7533709/PyMySQL-0.4-sock-sendall-fix-v2.patch that fixes all four occurrences of sock.send()
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Blocks: 669526
latest patch in prod.
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.