738514 - difficulty netbooting two mac minis host

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

Two of the mac minis out of the group of 80 that were just set up in scl3 refused to netboot at all.  I suspect that they may be configured for the wrong vlan, or it's a very good place to start looking, anyway.

The sticking point here is that the information in inventory (obtained from the csv that dmoore sent) does not correctly match physical location/asset with the host.

The asset tag and rack locations from physical inspection in the datacenter are:

8795     101-22 - SCL3 - 36.01
8824     101-22 - SCL3 - 35.01

However the hostnames/macs for these are:

host bld-lion-r5-018.try.releng.scl3-nic0  {
hardware ethernet 3c:07:54:72:77:c6;
fixed-address 10.26.64.38;
option host-name "bld-lion-r5-018.try.releng.scl3.mozilla.com";
}

host bld-lion-r5-029.try.releng.scl3-nic0  {
hardware ethernet 3c:07:54:72:51:b0;
fixed-address 10.26.64.49;
option host-name "bld-lion-r5-029.try.releng.scl3.mozilla.com";
}

Comment 1

[Derek Moore [:dmoore]]

I need a little clarification as to what needs to be fixed, here.

For example, I have the following in inventory:


Asset 8795
Rack 101-22
Location 36.01
MAC 3c:07:54:72:52:8a


Are you saying that when you put a monitor on that particular mini, it actually has the MAC 3c:07:54:72:77:c6?

Comment 2

[Amy Rich [:arr] [:arich]]

That's correct, they do not match (nor do at least some of the other machines; I don't know how many since I wasn't physically there).
(
The additional issue for the two specific MACs (3c:07:54:72:77:c6, 3c:07:54:72:51:b0) is that they got an IP from the dhcp server but were not able to netboot, so we're hoping that's just the wrong vlan.

Comment 3

[Derek Moore [:dmoore]]

I've confirmed we have some transposed asset tags. I'm auditing everything now, then we can discuss how to correct the situation in inventory.

Comment 4

[Derek Moore [:dmoore]]

Putting further investigation of this particular bug on hold until we have updated inventory.

Comment 5

[Derek Moore [:dmoore]]

Hopefully, the inventory correction has also fixed this bug. The ports were configured as a single contiguous block, so any remaining issues are likely a physical location/cabling issue.

Comment 6

[Jake Watkins [:dividehex]]

These hosts are still failing to netboot.  The ethernet cables should probably checked on both of these.

Comment 7

[Derek Moore [:dmoore]]

Thanks, Jake. We'll do a full physical inspection.

Comment 8

[Amy Rich [:arr] [:arich]]

08795 3c:07:54:72:57:d3 bld-lion-r5-020.try.releng.scl3
08824 3c:07:54:72:77:c6 bld-lion-r5-018.try.releng.scl3 


Both of these hosts are definitely on the network because I can ping them; so that's not the issue.

Comment 9

[Derek Moore [:dmoore]]

arr, does that eliminate any possible DCOps problem?

Comment 10

[Amy Rich [:arr] [:arich]]

dmoore: I think it unlikely that there are physical cable issues or issues with the switch config since the machines are reachable via ping and answer ssh connections (even though they do not have the correct pw since they haven't been correctly installed).  dividehex is going to investigate more today.

If it turns out that these need to go back to apple for service, I presume that this should get assigned to the SREs, correct?

Comment 11

[Jake Watkins [:dividehex]]

tl;dr:
Found a way to netboot and got minis imaged. There is an EFI update that may fix this.

Long:
I was able to netboot these 2 mac minis but not not all methods work.

Methods that failed:
Hold down N during boot
Issuing a bless command
Selecting the netboot image from the Startup under System Preferences

Methods that succeeded:
Holding down Option during boot and selecting the netboot image

I was also able to successfully netboot while holding down N once, but wasn't able to replicate this again.

When the netboot fails, the large flashing globe appears for a few seconds and then is replaced by the apple logo and the wait indicator (or log output if verbose boot logging is enabled.)  It will then boot from the disk.  From the view of the system logs on the bootpd server (install.build.releng.scl3.m.c) there may or may not be a few bootpd requests and there will be nothing from the tftpd.

The two hypotheses Amy and I came up with is, something was either dropping the bootpd packets or the iface on the mini was not initializing fast enough during the netboot process and therefore timed out and defaulted to boot the disk.

There is an EFI update that may address this issue but requires 10.7.3.  The current builder image is 10.7.2.  It might be possible to build a netboot set based on 10.7.3 and then have a deploy studio workflow install the update which is the same way we installed the EFI update on the scl1 r4 minis including the ones running snow.

Comment 12

[Jake Watkins [:dividehex]]

Forgot to include a link to the EFI update
http://support.apple.com/kb/DL1500

Comment 13

[Amy Rich [:arr] [:arich]]

020 is also showing a propensity for disappearing from the net/crashing/hanging.  If it does it again tomorrow, we should file a bug to have it serviced.

Comment 14

[Amy Rich [:arr] [:arich]]

So 20 went out for repairs, and I successfully reimaged the other two with the bless command this week.