846577 - Please rack and cable 50 ix chassis for linux and linux64 testing

Status: RESOLVED FIXED

Description

[Amy Rich [:arr] [:arich]]

We have 50 new iX chassis arriving at scl3 in the next couple weeks.  There should be stub entries for all hosts in DNS and inventory already. Please make sure that IPMI is configured and all hosts are set to boot from disk by default.

These hosts should be racked/cabled/VLANed as follows:

VLAN 240:

t-w864-ix-099.wintest.releng.scl3.mozilla.com
t-w864-ix-100.wintest.releng.scl3.mozilla.com


VLAN 256:

talos-linux32-ix-002.test.releng.scl3.mozilla.com
..
talos-linux32-ix-100.test.releng.scl3.mozilla.com


talos-linux64-ix-002.test.releng.scl3.mozilla.com
..
talos-linux64-ix-100.test.releng.scl3.mozilla.com


VLAN 216:

t-w864-ix-099-mgmt.inband.releng.scl3.mozilla.com
t-w864-ix-100-mgmt.inband.releng.scl3.mozilla.com

talos-linux32-ix-002-mgmt.inband.releng.scl3.mozilla.com
..
talos-linux32-ix-100-mgmt.inband.releng.scl3.mozilla.com


talos-linux64-ix-002-mgmt.inband.releng.scl3.mozilla.com
..
talos-linux64-ix-100-mgmt.inband.releng.scl3.mozilla.com

Comment 1

[Melissa O'Connor [:melissa]]

9 of these systems will arrive on 3/8, the remainder will arrive on 3/15.

If we can get the first 9 racked and ready to go quickly that would be great. :)

Comment 2

[Van Le [:van]]

The first 10 iX systems has been racked, cabled and IPMI configured. 

t-w864-ix-099-mgmt.inband.releng.scl3.mozilla.com is alive
t-w864-ix-100-mgmt.inband.releng.scl3.mozilla.com is alive
talos-linux64-ix-002-mgmt.inband.releng.scl3.mozilla.com is alive
..
talos-linux64-ix-039-mgmt.inband.releng.scl3.mozilla.com is alive

Comment 3

[Van Le [:van]]

switch ports tagged.

[e-mail]3# show | compare 
[edit interfaces interface-range win8]
-    member-range ge-0/0/0 to ge-0/0/36;
+    member-range ge-0/0/0 to ge-0/0/15;
-    member-range ge-1/0/0 to ge-1/0/36;
+    member-range ge-1/0/0 to ge-1/0/13;
[edit interfaces]
+   interface-range linux64-ix {
+       member-range ge-0/0/16 to ge-0/0/36;
+       member-range ge-1/0/14 to ge-1/0/36;
+       unit 0 {
+           family ethernet-switching {
+               port-mode access;
+               vlan {
+                   members test;
+               }
+           }
+       }
+   }
[edit vlans]
+   test {
+       vlan-id 256;
+   }

Comment 4

[Amy Rich [:arr] [:arich]]

t-w864-ix-100.wintest.releng.scl3.mozilla.com is not getting a DHCP lease.  Is there a cable or VLAN problem with it?

Comment 5

[Van Le [:van]]

Bad cable issue with t-w864-ix-100.wintest.releng.scl3.mozilla.com resolved.

Comment 6

[Derek Moore [:dmoore]]

Network hardware for the next three racks arrived today:

========

The EX4200s you ordered on PO # mrz-BCT-490-win8 have been shipped, see shipping information below:
FEDX STD OVR
752259237280
752259237279
752259237268
752259237257
752259237246
752259237290

They were delivered today, March 19, 2013.

The EX2200s and Rack Mounts on PO # mrz-BCT-489-win8 were also shipped, see shipping information below:
FEDX STD OVR
959690088260
959690088293
959690088308
959690088271
959690088282
959690088341
959690088330
959690088320
959690088319
959690088385
959690088374
959690088363
959690088352

These were also delivered today, March 19, 2013.

One item is currently backordered - EX-PWR-320-AC. We will send updated tracking information as soon as we have it for this item.

Comment 7

[Amy Rich [:arr] [:arich]]

Of the ones just installed, the problem children are:

talos-linux64-ix-044 - no IPMI
talos-linux64-ix-057 - WAY SLOW during the kickstart, power cycled, and got blinking cursor
talos-linux64-ix-060 - Reboot and Select proper Boot device
talos-linux64-ix-081 - no DHCP
talos-linux32-ix-008 - no IPMI

Comment 8

[Amy Rich [:arr] [:arich]]

All of these are working now except talos-linux64-ix-057 for which I opened bug 854597 to track that.

Comment 9

[Amy Rich [:arr] [:arich]]

The last batch should be in Friday or Monday.

Comment 10

[Melissa O'Connor [:melissa]]

DCOps how many arrived?

Comment 11

[Van Le [:van]]

10 arrived Monday, 13 remaining.

Comment 12

[Amy Rich [:arr] [:arich]]

As I sent in email, there should only be three chassis remaining based on the number of systems I already have info for.


I am unable to get to talos-linux32-ix-022-mgmt.inband.releng.scl3.mozilla.com

talos-linux32-ix-036-mgmt.inband.releng.scl3.mozilla.com had the wrong netmask (which I have fixed)

Comment 13

[Van Le [:van]]

My mistake, it looks like we've received 47 hosts so far.

10 ~ March 8
27 ~ March 25
10 ~ April 8

Pending a delivery of the remaining 3 chassis/12 hosts for
talos-linux32-ix-[088-100].test.releng.scl3.mozilla.com.


Let me know if there are any discrepancies.

Comment 14

[Van Le [:van]]

IPMI has been configured for the 10 chassis (40 nodes) we received on Monday.

talos-linux32-ix-049-mgmt.inband.releng.scl3.mozilla.com is alive
...
talos-linux32-ix-088-mgmt.inband.releng.scl3.mozilla.com is alive

Comment 15

[Amy Rich [:arr] [:arich]]

Problem hosts:

talos-linux32-ix-022-mgmt.inband.releng.scl3.mozilla.com - unpingable
talos-linux32-ix-085-mgmt.inband.releng.scl3.mozilla.com - can't get DHCP lease
talos-linux32-ix-086-mgmt.inband.releng.scl3.mozilla.com - can't get DHCP lease

Comment 16

[Melissa O'Connor [:melissa]]

The last 3 of these should have been delivered Friday.

Comment 17

[Melissa O'Connor [:melissa]]

The last 3 are in, we do not expect DCOps to rack them until next week.

Comment 18

[Van Le [:van]]

>talos-linux32-ix-022-mgmt.inband.releng.scl3.mozilla.com - unpingable
reconfigured.
talos-linux32-ix-022-mgmt.inband.releng.scl3.mozilla.com is alive

>talos-linux32-ix-085-mgmt.inband.releng.scl3.mozilla.com - can't get DHCP lease
>talos-linux32-ix-086-mgmt.inband.releng.scl3.mozilla.com - can't get DHCP lease
these 2 nodes are peculiar. eth0 is being seen as eth1 by the BIOS. Changed BIOS setting to PXE boot off eth1 instead of eth0. Kept cable in port that's supposed to be eth0 and updated inventory with new MAC. Let me know if there are any issues. They're both sitting at the OS option screen after PXE booting.

Comment 19

[Van Le [:van]]

>The last 3 are in, we do not expect DCOps to rack them until next week.

We decided to kick ass and get them all racked/cabled/inventoried anyways.

talos-linux32-ix-089-mgmt.inband.releng.scl3.mozilla.com is alive
...
talos-linux32-ix-100-mgmt.inband.releng.scl3.mozilla.com is alive

Please open individual bugs for any problematic hosts. These 50 iX servers are racked and cabled.

Comment 20

[Amy Rich [:arr] [:arich]]

You guys need to give me the csv for the rest of the machines.  They won't work until I put them in inventory with the k/v store info.

Comment 21

[Van Le [:van]]

csv sent to amy. please resolve bug if everything is copacetic.