441862 - Lots of machines assigned "static" IP addresses from the dynamic DHCP pool in the office

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

I'm connected to MozillaCorp and my computer got "dolske-laptop" as it's name.  Reed said to file a bug.

Comment 1

[Justin Dolske [:Dolske]]

I demand a new IP, this one is now tainted. :)

Comment 2

[matthew zeier [:mrz]]

What's the problem here? 

Comment 3

[Justin Dolske [:Dolske]]

I think the problem here is that the IP for dolske-laptop.office.mozilla.org is being handed out to a DHCP request from some other MAC address. If I then roll into the office, turn on my laptop, and try to ssh from my desktop to dolske-laptop, I'd be connecting to his box instead of mine.

Comment 4

[matthew zeier [:mrz]]

If Wil weren't onsite I'm guessing you just fail to connect!

What's your MAC address?  

Comment 5

[Justin Dolske [:Dolske]]

I've been using the MozillaCorp SSID without any problems (ok, well, without any DHCP problems! *cough*) since bug 438051 added these entries. I don't remember if I verified my laptop was getting the right IP, but it is right now:

IP 10.250.2.132, same MAC as in bug bug 438051 comment 0.

Maybe there's overlap between "my" IP and the pool used for random DHCP clients?

Comment 6

[matthew zeier [:mrz]]

Wil, what's your MAC and IP?

root@pinwheel (/var/named/office/) 3> grep dolske-laptop *
2.250.10.in-addr.arpa:132     IN      PTR     dolske-laptop.office.mozilla.org.
office.mozilla.org:dolske-laptop        IN A 10.250.2.132

 host dolske-laptop { hardware ethernet 00:17:f2:48:ff:e8; fixed-address 10.250.2.132;}

Comment 7

[Wil Clouser [:clouserw]]

My IP was 10.250.2.132 and my MAC is 00:17:f2:e9:1c:b6

I had to switch to Mozilla Guest though so my IP/name is different now.

Comment 8

[matthew zeier [:mrz]]

Wasn't aware that was possible - dave, does that ip need to be excluded from the pool?

Comment 9

[Dave Miller [:justdave]]

Yes, it does.  DHCP does look to see if an address is taken before handing it out, and dolke will get his IP if it isn't taken when he connects since his MAC address matches, but since it's in the pool, if dolke's not online at the time, the address may get assigned to someone else.  The fix is to give dolske an address outside of the pool range.

Comment 10

[Dave Miller [:justdave]]

And looking at the DHCP config, there's a whole mess of machines in addition to dolske's that are going to have this problem.  Right now, out of the 253 available IP addresses, we have 152 in the pool and 99 statically assignable.  Assuming there's open spaces in that 99 I'll shift some things around, but we might have to shrink the pool a little.

Comment 11

[Dave Miller [:justdave]]

er, make that 154 and 99, but you knew what I meant.

Comment 12

[Dave Miller [:justdave]]

The following machines are affected:

skittles      10.250.2.104
alarm-s       10.250.2.122
alarm-k       10.250.2.130
teal          10.250.2.131
dolske-laptop 10.250.2.132
dolske-n810   10.250.2.133
dolske-n800   10.250.2.135
backup        10.250.2.150
moz-mini      10.250.3.157
copier        10.250.2.181
copier-s      10.250.2.182
mcrprint      10.250.2.242

given that some of those are office utility servers, we probably better take a downtime window to fix this.

The following IPs have been reassigned effective immediately:

dolske-laptop 10.250.2.33
dolske-n810   10.250.2.34
dolske-n800   10.250.2.35

The rest I'll do sometime Saturday afternoon when nobody's around.

Comment 13

[matthew zeier [:mrz]]

not clear you can redo the alarm ones without reprogramming the alarm. 

Comment 14

[Dave Miller [:justdave]]

Hmm, the addresses in question appear to have been picked up via dynamic DHCP, and then the addresses they already had were just assigned.  That's kinda what this "feels" like.

I'm not sure the printers wouldn't have to be reprogrammed or anything either, so I guess this is better dealt with by Sean or someone in the office who has physical access to anything that might need other changes.

The IP range from 36 to 58 appears to be open still.

Comment 15

[Dave Miller [:justdave]]

Sean signed off on doing all of this except the alarms, he says the rest are all properly on DHCP and should just pick up the new IP when the current lease expires, so doing them any time long enough before the office opens for the leases to expire before people are in the office is fine.

We'll have to ask around about the alarm systems before messing with those.

Comment 16

[Justin Dolske [:Dolske]]

(In reply to comment #12)

> dolske-laptop 10.250.2.33
> dolske-n810   10.250.2.34
> dolske-n800   10.250.2.35

My N810 is getting .34, but DNS still says dolske-n810 is .133.

Comment 17

[Doug Turner (:dougt)]

ick!
dolske-n800:~ dougt$ hostname
dolske-n800.office.mozilla.org

Comment 18

[Dave Miller [:justdave]]

It's got the correct names/IPs in the zone files.  I just restarted the nameserver, try now?

Comment 19

[Dave Miller [:justdave]]

OK, I started to take care of the stuff listed in comment 12 and realized there's a whole bunch of mismatches in the zone files between the forward and reverse zones, and they really need to be audited so everything matches before anything else is done on IP reassignments.  Bunch of things in the forward lookup file that aren't listed in the reverse, lots of things that are in the build network in the reverse file and listed with an office network IP address in the forward file, etc.  I'll spin off a new bug for that.

In the meantime, I broke up the dynamic assignment ranges to exclude the statically-assigned addresses.  You can't specify exclusions, but it turns out you can specify more than one range, so there's just a bunch of ranges listed now that include everything except the ones that are taken.