Closed Bug 740633 Opened 8 years ago Closed 6 years ago

set up seamonkey HPs in scl3

Categories

(Infrastructure & Operations :: DCOps, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dustin, Unassigned)

References

Details

Attachments

(2 files)

These are up and in inventory and have iLO configured and everything!

https://inventory.mozilla.org/en-US/systems/racks/?location=19&status=&rack=183&allocation=

So I'll need to verify the iLO, and then get an OS installed on them.

Callek, how do you want to image these?  PuppetAgain?  If so, do you want to make one of them your puppet server?
Yea I think puppetAgain will be the best (shorter-term) solution for now, Some of these were/are slated for win systems at some point, and I don't expect to devote too much time to these too soon, but using one as a puppet master (at least temporarily) sounds great. (We may decide we want to make one of the VMs given to use the puppet master and move said VM to the HP, but we'll decide that later)
Depends on: 724968
Depends on: 744298
New bug 744298 is to get the OOB sorted out - the IPs in inventory appear to be incorrect.
No longer depends on: 724968
No longer blocks: 721516
Depends on: 721516
No longer depends on: 721516
This is just waiting on getting a public mirror of puppetagain (bug 769780, RSN) and a puppetmaster bootstrapping script (bug 769780) up and running.
I'm working on installing CentOS 6.3 on this now.  There's no kickstart, so it's taking a while.
Depends on: 783258
Need to change VLANs first - bug 783258
OK, sea-puppet is online with the usual root pw.

Callek, do you want to set the host up as a puppet master?
Assignee: dustin → bugspam.Callek
Assignee: bugspam.Callek → nobody
Component: Server Operations: RelEng → Release Engineering
Product: mozilla.org → SeaMonkey
QA Contact: arich
Version: other → unspecified
Assignee: nobody → bugspam.Callek
Depends on: 827079
Per e-mail convo with :dmoore, we're ready to move forward with this. ETA is up in the air though.

This will be the list of machines at https://inventory.mozilla.org/en-US/core/search/#q=sea-hp-linux64

Redundant power supplies are preferred, but not a requirement here. OOB/iLO connection is also preferred for IT's use, I don't have access to the OOB network to manage them though.

dmoore explicitly says: "I'm willing to let DCOps handle all of the IT-related items on your list. We'd generally punt for ifcfg changes, but my guys can handle a small change like that. In the interest of time, we can keep that inside DCOps instead of bouncing it around between teams."

The plan is as follows:

* [moco IT] Switch VLAN from community to moco-build-network
* [DCOps] Image the HP's using the Releng PuppetAgain CentOS6 base image, over a pxeboot/netboot. Inputting the *incorrect* puppetizing password.
* [moco IT] Switch VLAN from moco-build-network to community
* [moco IT] Ensure ifcfg is configured correctly for a static (not DHCP) IP/etc.
* [moco IT] reboot system one last time and hand back to Callek
* [SeaMonkey/Callek] connect systems to SeaMonkey's puppetmaster using the SeaMonkey password, and let the systems get setup automatically.
Assignee: bugspam.Callek → server-ops-dcops
Component: Release Engineering → Server Operations: DCOps
Product: SeaMonkey → mozilla.org
QA Contact: dmoore
Version: unspecified → other
(In reply to Justin Wood (:Callek) from comment #7)

> Redundant power supplies are preferred, but not a requirement here. 


I just wanted to point out that the DL120 platform only has a single power supply. We can deliver on everything else, but we can't upgrade to redundant power. Please keep that in mind when deploying.

We can split the servers evenly between different power feeds, if there's a way to design the setup to survive a loss of half the servers.
(In reply to Derek Moore [:dmoore] from comment #8)
> (In reply to Justin Wood (:Callek) from comment #7)
> 
> > Redundant power supplies are preferred, but not a requirement here. 
> 
> I just wanted to point out that the DL120 platform only has a single power
> supply. We can deliver on everything else, but we can't upgrade to redundant
> power. Please keep that in mind when deploying.

Ahh didn't realize that

> We can split the servers evenly between different power feeds, if there's a
> way to design the setup to survive a loss of half the servers.

If this requires *any* more effort that just plugging power cords into a different outlet within reach of the servers its more work than I find worth it for SeaMonkey here. Keep that in mind.
(In reply to Justin Wood (:Callek) from comment #9)

> If this requires *any* more effort that just plugging power cords into a
> different outlet within reach of the servers its more work than I find worth
> it for SeaMonkey here. Keep that in mind.

That's exactly what it means, but we should coordinate to make sure they are split in a meaningful fashion.
(In reply to Derek Moore [:dmoore] from comment #10)
> (In reply to Justin Wood (:Callek) from comment #9)
> 
> > If this requires *any* more effort that just plugging power cords into a
> > different outlet within reach of the servers its more work than I find worth
> > it for SeaMonkey here. Keep that in mind.
> 
> That's exactly what it means, but we should coordinate to make sure they are
> split in a meaningful fashion.

Half and half works for me, but feel free to adjust as suits your physical requirements of the space.
As far as redundancy, there are two non-replicated, critical hosts (puppet master and buildbot master) and the rest are all redundant to one another.  So there isn't actually an arrangement of power that will allow one side to be shut down without loss of functionality.  And even split just means half the machines to be recovered when one side goes down.
colo-trip: --- → scl3
:arr/:dustin, these hosts arent pxebooting to the same menu as the one in this mana page: https://mana.mozilla.org/wiki/display/DC/How+To+Reimage+Releng+iX+and+HP+Linux+Machines

the menu I see has only gives me the option to image CentOS MANUAL and :callek doesnt believe the necessary scripts will be installed.

do we need to setup sreg and a temporary IP in the correct vlan for it to image?
confirmed there is no /root/puppetize.sh installed
What VLAN are they in now?  They should be on 252 (https://inventory.mozilla.org/en-US/core/vlan/143/), which now that I think of it might not be plumbed to that rack, since it's in a different BU.
i was told to use 248 (releng-srv). i'll try 252 tomorrow.
That should be fine, too.  And that VLAN should have the same PXE options as any other VLAN, unless something has broken since last time I kickstarted a machine there.  Can you confirm it's getting an appropriate IP for that VLAN?  If so, can you capture a screenshot?
100% progress bar
this is where it's stuck at when i press 0 on the keyboard.
i got 1 working. ill work on the rest tomorrow.
done.

[vle@admin1a.private.scl3 ~]$ fping sea-hp-linux64-{2..13}.community.scl3.mozilla.com
sea-hp-linux64-2.community.scl3.mozilla.com is alive
sea-hp-linux64-3.community.scl3.mozilla.com is alive
sea-hp-linux64-4.community.scl3.mozilla.com is alive
sea-hp-linux64-5.community.scl3.mozilla.com is alive
sea-hp-linux64-6.community.scl3.mozilla.com is alive
sea-hp-linux64-7.community.scl3.mozilla.com is alive
sea-hp-linux64-8.community.scl3.mozilla.com is alive
sea-hp-linux64-9.community.scl3.mozilla.com is alive
sea-hp-linux64-10.community.scl3.mozilla.com is alive
sea-hp-linux64-11.community.scl3.mozilla.com is alive
sea-hp-linux64-12.community.scl3.mozilla.com is alive
sea-hp-linux64-13.community.scl3.mozilla.com is alive
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.