535839 - Search on preview is unavailable

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[krupa raj[:krupa]]

Search on preview is unavailable again.

https://preview.addons.mozilla.org/en-US/firefox/search?q=firefox&cat=all&advancedsearch=1&as=1&appid=1&lver=3.5&atype=0&pp=20&pid=3&sort=&lup=


HTTP Header-Server response:
HTTP/1.x 503 Service Unavailable

Server: Apache

Vary: Accept-Encoding

Cache-Control: public, max-age=3600

Content-Type: text/html; charset=UTF-8

Content-Encoding: gzip

Date: Fri, 18 Dec 2009 19:24:03 GMT

Expires: Fri, 18 Dec 2009 20:24:04 GMT

X-AMO-ServedBy: pm-app-amo24.mozilla.org

Via: Moz-Cache-zlb01

Connection: close

Last-Modified: Fri, 18 Dec 2009 19:24:04 GMT

X-Powered-By: PHP/5.2.9

X-Cache-Info: not cacheable; response code not cacheable

Comment 1

[Jeremy Orem [:oremj]]

Segfault:

#0  0x000000000041d30f in ReloadIndexSettings ()
#1  0x000000000041dd0f in CheckRotate ()
#2  0x000000000042c049 in ServiceMain ()
#3  0x00000031f3c1d994 in __libc_start_main () from /lib64/libc.so.6
#4  0x0000000000403839 in _start ()

Restarted searchd.

Comment 2

[krupa raj[:krupa]]

Search is once again unavailable

reason:HTTP/1.x 503 Service Unavailable

URL:https://preview.addons.mozilla.org/en-US/firefox/search?q=stephen3000!!!&cat=all&advancedsearch=1&as=1&appid=1&lver=3.5&atype=0&pp=20&pid=3&sort=&lup=

This is blocking QA from testing.Please fix.

Reopening....

Comment 3

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Trevor were you able to make sense of the last segfault?

I thought you also had a wrapper script that checks for searchd, is that not detecting these crashes?

I am also curious as to why this is (thankfully) not happening on production.

Comment 4

[Jeremy Orem [:oremj]]

Restarted the daemon again and it's back up.

Comment 5

[chizu]

(In reply to comment #3)
> Trevor were you able to make sense of the last segfault?
> 
> I thought you also had a wrapper script that checks for searchd, is that not
> detecting these crashes?
> 
> I am also curious as to why this is (thankfully) not happening on production.

Two different segfaults occur frequently. Production does crash but is protected from affecting the live site by the load balancer and restart script. Stage isn't load balanced. The wrapper script is occasionally failing to restart when a lock is in place during the crash.

To debug the segfaults I need to get symbols built and report them upstream. Fixing the upgrade to 0.9.9 bug would also be good, it might have fixed these bugs.

Comment 6

[chizu]

Debug symbols for 0.9.9 installed. Watching for any segfaults on preview.addons with the new version.

Comment 7

[chizu]

0.9.9-rc2 has continued to occasionally die but no segfaults from 0.9.9 yet.

Comment 8

[chizu]

Marking this solved. 0.9.9 hasn't crashed at all since the 7th.