902970 - wireless interface not showing up on talos-r4-lion-005.build.scl1.mozilla.com and talos-r4-lion-033.build.scl1.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Amy Rich [:arr] [:arich]]

talos-r4-lion-005.build.scl1.mozilla.com has been spitting out a lot of puppet errors trying to turn off the wireless interface on en1.  If you do an ifconfig -a, en1 doesn't even show up anymore, so I'm guessing we might have a hardware problem.  Do we have diags we can run on it to take a look?

Comment 1

[Vinh Hua [:vinh]]

Took host offline to run diagnostics.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Same on talos-r4-lion-033.

[[e-mail] ~]# networksetup -getairportpower en1
en1 is not a Wi-Fi interface.
** Error: Error obtaining wireless information.

yet system_profiler says

    Wi-Fi:

      Type: AirPort
      Hardware: AirPort
      BSD Device Name: en1
      Has IP Assigned: No
      IPv4:
          Configuration Method: DHCP
      IPv6:
          Configuration Method: Automatic
      Proxies:
          Exceptions List: *.local, 169.254/16
          FTP Passive Mode: Yes
      Service Order: 3


What's curious is, this hasn't affected the r5 builders running lion in scl3.  And that may actually be the clue: the image we're deploying to these r4 minis was built for the r5 minis.  I don't see why that would cause problems for some but not all hardware, though, so the diagnostics should continue.

Comment 3

[Vinh Hua [:vinh]]

Apple Hardware Test did not find any issues with the wifi card.

Comment 4

[Vinh Hua [:vinh]]

Host has been reimaged.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

And now,

[[e-mail] ~]# ifconfig en1
en1: flags=8823<UP,BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 1500
        ether 10:9a:dd:c0:61:b5 
        media: autoselect (<unknown type>)
        status: inactive

so, wtf, lion, wtf.

I'm reimaging it again to see if (a) en1 stays fixed and (b) maybe I can figure out bug 902903.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I rebooted - it should be back up and running shortly.

Comment 7

[Amy Rich [:arr] [:arich]]

033 is exhibiting the same issues.  Can you try powering it back on (I tried powering it off to see if a power cycle will fix it).

Comment 8

[Vinh Hua [:vinh]]

Lion-033 was in a powered off state. Should be accessible now.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

..and en1 is back.  From Amy's googling, it looks like this is just a bug in Lion. 

The thought is that we haven't seen it with the builders because the r5's have a different wifi chipset; the chips have different version numbers:

r4:

Wi-Fi:

      Software Versions:
          CoreWLAN: 2.1.1 (211.3)
          CoreWLANKit: 1.0.1 (101.1)
          Menu Extra: 7.0.1 (701.2)
          configd plug-in: 7.1.1 (711.1)
          System Profiler: 7.0 (700.3)
          IO80211 Family: 4.1.1 (411.1)
          WiFi Diagnostics: 1.0.1 (101.1)
          AirPort Utility: 5.5.3 (553.20)
      Interfaces:
        en1:
          Card Type: AirPort Extreme  (0x14E4, 0x93)
          Firmware Version: Broadcom BCM43xx 1.0 (5.100.98.75.18)
          MAC Address: 10:9a:dd:c0:2c:7f
          Locale: FCC
          Country Code: X0
          Supported PHY Modes: 802.11 a/b/g/n
          Supported Channels: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 36, 40, 44, 48, 52, 56, 60, 64, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 149, 153, 157, 161, 165
          Wake On Wireless: Supported
          AirDrop: Supported
          Status: Off

r5:

Wi-Fi:
      
      Software Versions:
          CoreWLAN: 2.1.1 (211.3)
          CoreWLANKit: 1.0.1 (101.1)
          Menu Extra: 7.0.1 (701.2)
          configd plug-in: 7.1.1 (711.1)
          System Profiler: 7.0 (700.3)
          IO80211 Family: 4.1.1 (411.1)
          WiFi Diagnostics: 1.0.1 (101.1)
          AirPort Utility: 5.5.3 (553.20)
      Interfaces:
        en1:
          Card Type: AirPort Extreme  (0x14E4, 0xE4)
          Firmware Version: Broadcom BCM43xx 1.0 (5.100.98.75.18)
          MAC Address: 7c:c3:a1:73:d3:67
          Locale: FCC
          Country Code: X0
          Supported PHY Modes: 802.11 a/b/g/n
          Supported Channels: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 36, 40, 44, 48, 52, 56, 60, 64, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 149, 153, 157, 161, 165
          Wake On Wireless: Supported
          AirDrop: Supported
          Status: Off

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

We'll open new bugs for any other scl1 hosts that need a hard power cycle.

Comment 11

[Armen [:armenzg]]

What should we do with the hosts?
Can we put them straight into production?

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

If the wifi's working, yes.