socorro staging elastic search instance unreachable

RESOLVED INVALID

Status

RESOLVED INVALID
6 years ago
5 years ago

People

(Reporter: lars, Unassigned)

Tracking

Details

(Reporter)

Description

6 years ago
socorro's staging is supposed to have an instance of elastic search running.  It is either missing or unreachable:

2012-06-19 14:08:38,399 CRITICAL - Thread-7 - Submition to Elastic Search failed for 4ff1179c-a02c-4f35-b394-9dfda2120619
2012-06-19 14:08:38,400 CRITICAL - Thread-7 - Caught Error: <class 'urllib2.URLError'>
2012-06-19 14:08:38,401 CRITICAL - Thread-7 - <urlopen error [Errno 111] Connection refused>

the submission url that we're using is: http://hp-node61.phx1.mozilla.com:9999/queue/tasks/

Comment 1

6 years ago
Can we either fix it or reconfigure stage to not use it?  We're not shipping it this week, but having the config be broken blocks us from QA and therefore shipping today.
Severity: normal → critical

Updated

6 years ago
Assignee: server-ops → rbryce

Comment 2

6 years ago
There was an upgrade to the SocorroSearchService just yesterday: https://bugzilla.mozilla.org/show_bug.cgi?id=766246. Relevant, perhaps?

I see many ports open by java in netstat... what port/service is it that's down? My knowledge of socorro and elastic search is very limited.

Comment 3

6 years ago
With help from #breakpad, we have disabled the processors from trying to use Elastic Search. This is in modules/webapp/files/socorro-stage/etc-socorro/socorro-processor.conf. When whatever the underlying problem is fixed, just uncomment the line and puppet should take care of the rest.

The two nodes affected are socorro-processor(1|2).stage.metrics.phx1.

Comment 4

6 years ago
jakem 

Im not sure what needs to happen next.  I assigned this to myself while oncall to stop paging.  At this point Im not sure I should be the assignee

Updated

6 years ago
Severity: critical → normal

Updated

6 years ago
Assignee: rbryce.bugs → server-ops-webops
The URL used in the processors was wrong (should be 10.8.81.222:9200/queue/tasks/ according to https://mana.mozilla.org/wiki/display/websites/Socorro+Search+Service#SocorroSearchService-LoadBalancingCaching ). We will try again that soon and reopen bugs if needed. 

Thanks for your help, folks!
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → INVALID
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.