530562 - [AMO] searchd is dead on preview.amo

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

From preview.addons.mozilla.org:

PHP Fatal error:  Uncaught exception 'AddonsSearchException' with message 'could not connect to searchd' in /data/www/addons.mozilla.org-preview/site/vendors/sphinx/addonsSearch.php:214

Chizu - is this emailing you when it dies?  Is there a reason it seems to die so often?

Comment 1

[Shyam Mani [:fox2mike]]

I just kicked searchd on pm-app-sphinx01 and 02 should be back up now. I'll leave this open for chizu to figure the rest of it out.

Comment 2

[Shyam Mani [:fox2mike]]

Yup, the URL in the bug = workie for me now.

Comment 4

[Shyam Mani [:fox2mike]]

Trevor, can you see if anything failed? or if you can figure out what happened?

Comment 5

[Shyam Mani [:fox2mike]]

FWIW, the logs have nothing really...so I guess there is nothing further to dig. It stops on the 19th and starts again when I kicked searchd.

I've asked Dave Dash to turn on some debug stuff in case this happens again so we might have something useful in the logs.

Comment 6

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Tracking the general issue of crashiness on preview in bug 530830.

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; it's been back up for a while, now.

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

Down again (been down for a couple hours, at least).  Any chance we can have this monitored in nagios?  It does affect our testing schedule, especially closer to release.  Thanks!

(Dave/Krupa told me to reopen; if you'd rather a new bug be filed, let me know, thanks!)

Comment 9

[chizu]

It is monitored in nagios. Looking now.

Comment 10

[chizu]

It didn't crash this time. It did exit on both boxes, which is probably a new issue. I'll keep looking into it.

Comment 11

[krupa raj[:krupa]]

Search is down again and it is blocking QA as we have a release on 2/2

Comment 12

[chizu]

I have this tracked down to the update and rotate scripts occasionally running into each other. They now run further apart and lock to prevent conflicts. On-call is paged and has instructions for restarting it if something else goes wrong.

Comment 13

[Stephen Donner [:stephend] Not actively reading bugmail]

Thx, Trevor; now working again: https://preview.addons.mozilla.org/en-US/firefox/search?q=foxytunes&cat=all&advancedsearch=1&as=1&appid=1&lver=any&atype=0&pp=20&pid=5&sort=&lup=