602196 - bm-foopy.build.mozilla.org and bm-remote.build.mozilla.org are offline

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Mike Taylor [:bear]]

moz:~ bear$ dig bm-foopy.build.mozilla.org

;; QUESTION SECTION:
;bm-foopy.build.mozilla.org.	IN	A

;; ANSWER SECTION:
bm-foopy.build.mozilla.org. 300	IN	CNAME	foopy.build.mtv1.mozilla.com.
foopy.build.mtv1.mozilla.com. 300 IN	A	10.250.48.9

Comment 1

[Aki Sasaki (not active)]

We'll need these for Android testing.
We're already way behind on this, so this is much appreciated!

Comment 2

[Aki Sasaki (not active)]

(I can get to the 3 minis in bug 550357 via http + ssh, but not the load balancer bm-remote.b.m.o via http)

Comment 3

[Phong Tran [:phong]]

rebooted bm-foopy.build.  I don't know where bm-remote.build is.  I can't find it in inventory.

Comment 4

[Aki Sasaki (not active)]

It's a load balancer in front of the three minis in bug 550357, definitely in MV but not sure where.

It's less urgent than foopy, however; thanks for getting that back up :)

Comment 5

[Justin Dow [:jabba]]

It looks like bm-remote has been down since the last power outage, because the load balancer didn't have the configuration saved properly. I reconfigured it and saved the configuration. It is up again.

Comment 6

[Mike Taylor [:bear]]

odd, I can do a dig on bm-remote.b.m.o but when I try to ping it I get this:

moz:~ bear$ dig bm-remote.build.mozilla.org

;; Query time: 94 msec
;; SERVER: 10.2.74.123#53(10.2.74.123)
;; WHEN: Wed Oct  6 18:03:06 2010
;; MSG SIZE  rcvd: 206

moz:~ bear$ ping bm-remote.build.mozilla.org
ping: cannot resolve bm-remote.build.mozilla.org: Unknown host

Comment 7

[Justin Dow [:jabba]]

I think that might be VPN playing games with you. I have that frequently with Viscosity VPN client, where userspace programs like ping and ssh will use Viscosity's custom resolver which tries to use the dns servers from your local network as well as the vpn network, and dig, host, nslookup etc. will just query whatever dns server is in /etc/resolv.conf ... at least that is what I think is happening and restarting your vpn client, computer, flushing dns cache, or something along those lines might work. :)

Comment 8

[Mike Taylor [:bear]]

well, restarting vpn client, computer, flushing dns cache or something... to be blunt would require me setting up my test environment of multiple vm's tegra's, virtualenv and other tools such that I would lose close to an hour each time.

so not an option

what are the dns servers I need to add to my resolve.conf to prevent this - I plan on running build-vpn full time so can add them to my IP config

Comment 9

[Justin Dow [:jabba]]

I think 

nameserver 10.2.74.125
nameserver 10.2.74.127

would be correct, and also in Viscosity preferences (if that is what you are using) uncheck the option called "use alternate DNS support". Hopefully that works.

Comment 10

[Mike Taylor [:bear]]

thanks - i'll tweak the dns setup.  at least now the resolv.conf when build-vpn is active actually contains a mozilla name server :)

/me moves on to next "oh my lord, we gotta fix this" item

thanks for the server restarts!