Closed Bug 748072 Opened 12 years ago Closed 12 years ago

Move all the Eideticker machines in MV to haxxor

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cmtalbert, Assigned: dividehex)

Details

In order to keep the phones associated with the Eideticker machines online, we should put these machines in Haxxor.  We tried housing the existing MV Eideticker machine on the MV Office floor, but we've had no end of connectivity issues with the phone remaining on the MV Office wireless network.  

By putting these three boxes in Haxxor, we can have the phones be on the RF network in there and they should be much more reliable.  I'd like to move the three machines (the two new ones and the one existing one) into Haxxor either this afternoon (Monday, 4/23) or tomorrow.

I'm happy to do the actual lifting and moving and wiring, I just need a advice regarding plugging the ethernet cable from the eideticker machines into the proper switch in that room (there are several switches and I don't know if it matters which slot things are plugged into).
Our space and power in haxxor is very limited due to all of the tegras in there. All of the switches are on the releng network to my knowledge, and that may not be the right network for you.   What sort of hardware (boxes) are we talking about (what size, power requirements, etc), and how many phones?  Do these hosts have inventory entries?  Is there a reason we aren't putting these machines in one of the machine rooms instead?
Hi Amy, let me address each concern.

(In reply to Amy Rich [:arich] [:arr] from comment #1)
> Our space and power in haxxor is very limited due to all of the tegras in
> there. 
Yes, but over a quarter of the room is filled with boxes of random stuff (tegra dev boards, surge protectors etc). I think those boxes could be easily moved out and we'd have plenty of room for these three machines. Jake and I talked about this, and I've offered to help move the boxes.

The machines are small mini-towers, they are Dell t1600's: http://www.dell.com/downloads/global/products/precn/en/precision-t1600-spec-sheet.pdf

> All of the switches are on the releng network to my knowledge, and
> that may not be the right network for you.
They can be on the releng network, as long as they are accessible from the MV office network. This is important because this system will be used for our developers to log into a webservice hosted on one of these machines in order to run development builds through the Eideticker performance analysis. (sort of like a try server system).

>   What sort of hardware (boxes)
> are we talking about (what size, power requirements, etc), and how many
> phones?
I linked the pdf of the machines specs up above. The specific configuration of the machine can be found in the attachment on bug 745045 (https://bugzilla.mozilla.org/attachment.cgi?id=614637). For each box there is one phone.  The phones are connected to the HDMI inputs of the blackmagic video capture cards, and this is why there has to be a 1:1 mapping between box and phone.  When it's all said and done, we'll have three dell boxes and three phones in haxxor as part of the eideticker setup.

> Do these hosts have inventory entries?
No. Not yet. As you can see on bug 745045, there was some confusion around delivery and the real machines are expected later today.

>  Is there a reason we aren't
> putting these machines in one of the machine rooms instead?
Yes, because the phones that these machines are attached to need to be on an open wifi network, and we've had a ton of problems keeping our phones on the network in the mountain view office (where we were originally staging the eideticker box). As I understand it, haxxor was built for phone automation specifically to reduce the problems we have keeping phones on wifi, and so that's why I'm proposing putting these systems in there.  The tegras do not use wifi and should be moved out of haxxor at some point in favor of real phone automation (because we'll be doing more automation using real phones that need a shielded and open wifi network as this year progresses).

The T1600 boxes cannot be on wifi themselves as they have no wireless cards, which is why I brought up the problem with needing to plug them into switches.
There is a lot of boxxed stuff in there that's releng specific, but we don't really have anyplace else to put it that's secure.  I think Jake rearranged some things so there's a small bit of space for now.  Once the scl3 expansion is done, that room should be virtually empty, but we're in close quarters for now.

The releng network is not available from the mv office unless one is on the releng vpn, but we might be able to set up a couple switch ports that are trunked to the desktop vlan (I'm not sure if those switches are even connected to the desktop vlan).
dustin might have some insight there (do we have other hosts on the desktop network in there?)

The machines will need inventory and dns entries before we can put them on the network, though.  Please work with jake to get the machines (mini towers and phones) in inventory and installed into the room.
Assignee: server-ops-releng → jwatkins
Yes, those switches are managed and I can plumb the desktop VLAN in there.

We can make this work - in fact, this is a better use of a faraday cage than (hardwired) tegras.  That said, thank you for understanding our general aversion to having non-production-ready, unmanaged systems in our facilities (and particularly on the releng networks).
dustin: can you please change the follow ports to the desktop vlan?

switch2.df202-2.ops.mtv1:43
switch2.df202-2.ops.mtv1:44
switch2.df202-2.ops.mtv1:45
(In reply to Amy Rich [:arich] [:arr] from comment #3)
> The machines will need inventory and dns entries before we can put them on
> the network, though.  Please work with jake to get the machines (mini towers
> and phones) in inventory and installed into the room.
Awesome, thanks for the help. I'll post a comment with all the asset numbers and MAC addresses and hostnames once I have all the systems in one place. And I can pull all the info off of them.
One potential sticking-point here is the wifi in haxxor.  If that's piped into the releng network, we'll need to figure out how to get these phones on a different network, since this stuff can't be on the releng network.
(In reply to Jake Watkins [:dividehex] from comment #5)
> dustin: can you please change the follow ports to the desktop vlan?

vlan 200 
   name "corp" 
   untagged 43-45

(and it's properly trunked)
Ok, great. I've got the last of the software installing on these boxes now.

Here are the asset tags/mac addresses and some first guesses at hostnames:
= ateam.eideticker-galnex-1 =
mac: a0:0b:ba:da:88:45
asset: 09800

= ateam.eideticker-g2x-1 =
mac: 78:d6:f0:a6:e6:71
asset: 08378

= ateam.eideticker-g2x-2 =
mac: 78:d6:f0:f6:6f:eb
asset: 10501

= ateam.eideticker-1 =
mac: 18:03:73:e5:75:42
asset: 7940

= ateam.eideticker-2 =
mac: 78:2b:cb:9a:48:34
asset: 05336

= ateam.eideticker-3 =
mac: d4:be:d9:a6:67:38
asset: 09766

Since the two of the machines are *still* installing the android-sdk I will physically move these guys in tomorrow (Friday).
ateam-eideticker-galnex-1   IN  A       10.250.1.179
ateam-eideticker-g2x-1      IN  A       10.250.1.180
ateam-eideticker-g2x-2      IN  A       10.250.1.189
ateam-eideticker-1          IN  A       10.250.1.191
ateam-eideticker-2          IN  A       10.250.1.212
ateam-eideticker-3          IN  A       10.250.1.252

All are added to DNS and DHCP.  I didn't do inventory.
I wrote up some basic systems documentation here:
  https://mana.mozilla.org/wiki/display/websites/Eideticker

Clint, can you have a look and update where I've gotten things wrong, or where I don't know?

I'm working with ravi to figure out the wireless-security questions.
Added to inventory:

ateam-eideticker-galnex-1
ateam-eideticker-g2x-1
ateam-eideticker-g2x-2
ateam-eideticker-1
ateam-eideticker-2
ateam-eideticker-3

Still need serial #s
(In reply to Dustin J. Mitchell [:dustin] from comment #11) 
> I'm working with ravi to figure out the wireless-security questions.
For wireless, can we put the phones onto the 10.250.50.xxx subnet that we are using for the autophone phones? (bug 721482).

Those are accessible from the office VLAN so there would be no problem to have them on the same network. Also, as far as I can tell none of the releng tegras are on the 10.250.50.xxx network, so it should be pretty safe.
(In reply to Jake Watkins [:dividehex] from comment #12)
> Added to inventory:
> 
> ateam-eideticker-galnex-1
> ateam-eideticker-g2x-1
> ateam-eideticker-g2x-2
> ateam-eideticker-1
> ateam-eideticker-2
> ateam-eideticker-3
> 
> Still need serial #s
From the phones as well as the machines?
(In reply to Clint Talbert ( :ctalbert ) from comment #13)
> (In reply to Dustin J. Mitchell [:dustin] from comment #11) 
> > I'm working with ravi to figure out the wireless-security questions.
> For wireless, can we put the phones onto the 10.250.50.xxx subnet that we
> are using for the autophone phones? (bug 721482).

That's the releng network, so using that network takes advantage of two bugs that we should fix ASAP:
 * physical access to the releng network
 * firewall access to the releng network from the desktop network

> Those are accessible from the office VLAN so there would be no problem to
> have them on the same network. Also, as far as I can tell none of the releng
> tegras are on the 10.250.50.xxx network, so it should be pretty safe.

10.250.50/24 is the same subnet as 10.25.48.0 (it's a /22 IIRC), so this *is* a problem.

Let's get these set up, then figure out what to do about the wireless on a new bug.

(In reply to Clint Talbert ( :ctalbert ) from comment #14)
> From the phones as well as the machines?

Yes, plz
Adding in serial numbers:
ateam-eideticker-3(In reply to Clint Talbert ( :ctalbert ) from comment #9)
= ateam.eideticker-galnex-1 =
mac: a0:0b:ba:da:88:45
asset: 09800
serial: R21C12P340P
Switch Ports: n/a (wireless)
Date Purchased: April 2012
OS: Android 4.0.2 

= ateam.eideticker-g2x-1 =
mac: 78:d6:f0:a6:e6:71
asset: 08378
Serial: 105KPRW33142
Switch Ports: n/a (wireless)
Date Purchased: November 2011?
OS: Android 2.3.3

= ateam.eideticker-g2x-2 =
mac: 78:d6:f0:f6:6f:eb
asset: 10501
serial: 112KP2K525413
Switch Ports: n/a (wireless)
Date Purchased: April 2012
OS: Android 2.3.3
 
= ateam.eideticker-1 =
mac: 18:03:73:e5:75:42
asset: 7940
serial: 00186-141-490-018
Switch Ports: Switch: SW2 DF202-2, port: 43
Date Purchased: October 2011?
OS: Ubuntu 11.10 64 bit
 
= ateam.eideticker-2 =
mac: 78:2b:cb:9a:48:34
asset: 05336
serial: 00186-104-283-954
Switch Ports: Switch: SW2 DF202-2, port: 44
Date Purchased: April 2012
OS: Ubuntu 11.10 64 bit 

= ateam.eideticker-3 =
mac: d4:be:d9:a6:67:38
asset: 09766
serial: 00186-159-953-483
Switch Ports: Switch: SW2 DF202-2, port: 45
Date Purchased: April 2012
OS: Ubuntu 11.10 64 bit
Thanks Clint.  I have updated inventory with this info.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.