Closed Bug 779912 Opened 12 years ago Closed 12 years ago

[prod] sp-admin01 cannot run hadoop jobs

Categories

(Infrastructure & Operations Graveyard :: NetOps: DC ACL Request, task, P1)

Product:
Infrastructure & Operations Graveyard
Component:
NetOps: DC ACL Request
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rhelmer, Assigned: cransom)

References

Details

Looks like sp-admin01 can't run hadoop jobs:

12/08/01 17:00:31 INFO ipc.Client: Retrying connect to server: hp-admin01.phx1.mozilla.com/10.8.101.205:8020. Already tried 0 time(s).
12/08/01 17:00:52 INFO ipc.Client: Retrying connect to server: hp-admin01.phx1.mozilla.com/10.8.101.205:8020. Already tried 1 time(s).
12/08/01 17:01:13 INFO ipc.Client: Retrying connect to server: hp-admin01.phx1.mozilla.com/10.8.101.205:8020. Already tried 2 time(s).
12/08/01 17:01:34 INFO ipc.Client: Retrying connect to server: hp-admin01.phx1.mozilla.com/10.8.101.205:8020. Already tried 3 time(s).

Also I see that there isn't anything on https://crash-analysis.mozilla.com/crash_analysis/modulelist/ newer than 2012-06-20, I thought there was a nagios job watching this directory?
Just want to clarify that this is a production issue.
Severity: normal → major
Summary: sp-admin01 cannot run hadoop jobs → [prod] sp-admin01 cannot run hadoop jobs
Perhaps an ACL was modified or removed ? 

[tmary@sp-admin01.phx1 ~]$ nc -vv  -w 5 hp-admin01.phx1.mozilla.com 8020
nc: connect to hp-admin01.phx1.mozilla.com port 8020 (tcp) timed out: Operation now in progress


--
Netops, can you verify the ACL in comment 2?
Assignee: server-ops → network-operations
Component: Server Operations → Server Operations: ACL Request
QA Contact: jdow → ravi
Keeping priority but stopping the bug from paging oncall.
Severity: major → normal
Priority: -- → P1
Assignee: network-operations → cransom
There's no reference to 10.8.101.205 in the firewall, did that change recently?  The addresses we have for the hp-admin boxes are:
address hp-admin01a 10.8.101.201/32;
address hp-admin01b 10.8.101.202/32;
address hp-admin02a 10.8.101.203/32;
address hp-admin02b 10.8.101.204/32;
I added hp-admin01 to the list.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Blocks: 765001
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.