914570 - DNS is broken in scl1 (buildapi giving 502 bad gateway errors)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Ed Morley [:emorley]]

[12:26:17.595] GET https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-central/rev/be1053dc223b?format=json [HTTP/1.1 502 Bad Gateway 718ms]

Breaks TBPL retriggers/cancellation etc.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

10.12.48.19 (ns-vip.build.scl1.mozilla.com) isn't responding to DNS requests. From nagios there is also an ntp problem (ntp.build.mozilla.org). We should check DHCP too.

Comment 2

[Ludovic Hirlimann [:Usul]]

(In reply to Nick Thomas [:nthomas] from comment #1)
> 10.12.48.19 (ns-vip.build.scl1.mozilla.com) isn't responding to DNS
> requests. From nagios there is also an ntp problem (ntp.build.mozilla.org).
> We should check DHCP too.

dhcp checked on both admin nodes.

Comment 4

[Ed Morley [:emorley]]

Just mentioning in case people haven't seen the duped bug - this bug means that both completed, running and pending jobs are not showing up on TBPL, and as such all main trees are closed.

Comment 5

[Ludovic Hirlimann [:Usul]]

ssh -l root ns-vip.build.scl1.mozilla.com
ssh: connect to host ns-vip.build.scl1.mozilla.com port 22: No route to host

I can't find anything on this host in Inventory. Nor in mana : https://mana.mozilla.org/wiki/dosearchsite.action?queryString=ns-vip.build.scl1.mozilla.com

Comment 6

[Ludovic Hirlimann [:Usul]]

Thanks to arr we kicked keepalived.

Comment 7

[Amy Rich [:arr] [:arich]]

An outage report will be forthcoming with more details.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

RelEng infra is all fixed up.

Comment 9

[Ed Morley [:emorley]]

I've filed bug 914699 for a nagios alert for self-serve that emails sheriffs@

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Bug 914735 added nagios checks for the DNS vip in SCL1. Previously we only had them on the two admin machines behind the vip.