766303 - socorro staging elastic search instance unreachable

Status: RESOLVED INVALID

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[K Lars Lohn [:lars] [:klohn]]

socorro's staging is supposed to have an instance of elastic search running.  It is either missing or unreachable:

2012-06-19 14:08:38,399 CRITICAL - Thread-7 - Submition to Elastic Search failed for 4ff1179c-a02c-4f35-b394-9dfda2120619
2012-06-19 14:08:38,400 CRITICAL - Thread-7 - Caught Error: <class 'urllib2.URLError'>
2012-06-19 14:08:38,401 CRITICAL - Thread-7 - <urlopen error [Errno 111] Connection refused>

the submission url that we're using is: http://hp-node61.phx1.mozilla.com:9999/queue/tasks/

Comment 1

[Laura Thomson :laura]

Can we either fix it or reconfigure stage to not use it?  We're not shipping it this week, but having the config be broken blocks us from QA and therefore shipping today.

Comment 2

[Jake Maul [:jakem]]

There was an upgrade to the SocorroSearchService just yesterday: https://bugzilla.mozilla.org/show_bug.cgi?id=766246. Relevant, perhaps?

I see many ports open by java in netstat... what port/service is it that's down? My knowledge of socorro and elastic search is very limited.

Comment 3

[Jake Maul [:jakem]]

With help from #breakpad, we have disabled the processors from trying to use Elastic Search. This is in modules/webapp/files/socorro-stage/etc-socorro/socorro-processor.conf. When whatever the underlying problem is fixed, just uncomment the line and puppet should take care of the rest.

The two nodes affected are socorro-processor(1|2).stage.metrics.phx1.

Comment 4

[Rick Bryce [:rbryce]]

jakem 

Im not sure what needs to happen next.  I assigned this to myself while oncall to stop paging.  At this point Im not sure I should be the assignee

Comment 5

[[DEACTIVATED] Adrian Gaudebert]

The URL used in the processors was wrong (should be 10.8.81.222:9200/queue/tasks/ according to https://mana.mozilla.org/wiki/display/websites/Socorro+Search+Service#SocorroSearchService-LoadBalancingCaching ). We will try again that soon and reopen bugs if needed. 

Thanks for your help, folks!