567409 - SUMO is reporting "Too many connections"

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[James Socol [:jsocol, :james]]

support.mozilla.com is throwing "too many connections" database errors in production over HTTPS (HTTP seems to be comfortably bouncing off Zeus for now).

Comment 1

[Dave Miller [:justdave]]

databases are swamped with hundreds of
SELECT COUNT(*) AS count FROM tiki_comments a where a.`object`='1' and a.`objectType` = 'forum' and a.`commentDate` > 1273258041

with random dates on the end.

Looks like someone's scraping the forums if I had to guess.

Comment 2

[Paul Craciunoiu [:paulc]]

Do you have some thoughts on what would be a good approach here? Looks like caching doesn't help?

I believe that query is fairly contained so we could temporarily patch the code. It would report the wrong number of pages however. Is there a better way?

Comment 3

[Paul Craciunoiu [:paulc]]

Just had a thought - would it help if the date was fixed? We could easily do that, at least for today.

Comment 4

[Jeremy Orem [:oremj]]

Blocked google bot. Should be getting better now.  Let me know if you want to patch that date.

Comment 5

[Paul Craciunoiu [:paulc]]

I pasted this over IRC: http://paulc.pastebin.mozilla.org/726684
It should help with the hits updates. Patched locally and didn't see any issues, so if the load is still too high and this would really help, go for it.

Comment 6

[Paul Craciunoiu [:paulc]]

(In reply to comment #4)
> Blocked google bot. Should be getting better now.  Let me know if you want to
> patch that date.
I would rather not, unless you think it would help a lot. I have doubts that it would, since the count AFTER a date (be it fixed or not) will not be constant.

Comment 7

[James Socol [:jsocol, :james]]

(In reply to comment #1)
> Looks like someone's scraping the forums if I had to guess.

I had Cheng turn off his sumobot scraping for the weekend. That should help, too.

Comment 8

[Jeremy Orem [:oremj]]

[root@mradm02 support.mozilla.com]# curl -s 'http://paulc.pastebin.mozilla.org/?dl=726684' | patch -p0
(Stripping trailing CRs from patch.)
patching file webroot/lib/commentslib.php
patch unexpectedly ends in middle of line
Hunk #2 succeeded at 1333 with fuzz 2.