There was a sudden jump in rows in the django_session table around Apr 4th-5th. We averaged around 5MM rows, but are suddenly up to 33MM today. There are no sessions older than today (daily GC is working) and the sessions all appear to contain the same data, which decoded, is essentially "testcookie=worked". You (andym) have done the most investigation on this. Let's hope your current theory about django's session deleter being broken is right. :) Thanks for investigating. Let IRC know if you need help, I'm sure db/ops is happy to help with anything to fix their swapping servers. :)
It's not about us being happy that "our swapping servers are fixed", it's that everyone, devs and ops, will be happy when the addons services do not fall over due to this extra load. This is an extremely important problem we are catching before it brings servers down.
https://github.com/mozilla/zamboni/commit/806c84 Testing out signed cookies, so we don't need db sessions at all.
See also bug 860509
Awesome! Be sure to let us know how it goes. :)
It was hairy, has the db traffic dropped?
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You've already deployed it to production?
Massively better. Just so you can see for yourself. :) https://graphite-phx1.mozilla.org/render/?width=586&height=308&_salt=1365615431.75&from=00%3A00_20130402&target=hosts.addons5_db_phx1_mozilla_com.disk.sda2.disk_ops.read&target=hosts.addons5_db_phx1_mozilla_com.disk.sda2.disk_ops.write&target=hosts.addons6_db_phx1_mozilla_com.disk.sda2.disk_ops.write&target=hosts.addons6_db_phx1_mozilla_com.disk.sda2.disk_ops.read&target=hosts.addons4_db_phx1_mozilla_com.disk.sda2.disk_ops.read&target=hosts.addons4_db_phx1_mozilla_com.disk.sda2.disk_ops.write
You need to log in before you can comment on or make changes to this bug.