Closed Bug 661010 Opened 13 years ago Closed 13 years ago

figure out how to clone win64-ix-ref

Categories

(Infrastructure & Operations :: RelOps: General, task, P3)

Product:
Infrastructure & Operations
Component:
RelOps: General
Platform:
x86
macOS
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: armenzg, Assigned: mlarrain)

References

Details

zandr had to re-image the machine in bug 645024.

If I recreate the machine we can get a consultant he has lined up to try to clone it with other different tool than CloneZilla.

The IP seems to be 10.250.49.173 and we have to figure out the IPMI address and if RDP is enabled.
Depends on: 661015
Host is up @ 10.250.49.239
Host is running through a ghost image to an attached USB drive.  Once this finishes, we'll try to reimage.
image backup done.  need a machine i can try to restore to.

ghost iso is @ \\fs2\IT\Microsoft\NGH1501_AllWin_English_SrdAndProd.iso .
From talking with mrz; this bug is tracking making a copy of a vanilla install of win2008 (64bit). 

The machine being copied *from* does not have any of Armen's work on it.

The purpose of this experiment is only to see if IT have tools that can take a working image from one win2008 machine, and put it onto another blank machine which boots correctly.
Depends on: 645024
(In reply to comment #4)
> image backup done.  need a machine i can try to restore to.
> 
> ghost iso is @ \\fs2\IT\Microsoft\NGH1501_AllWin_English_SrdAndProd.iso .

mv-moz2-linux-ix-slave22 is in 650castro, is idle and can be reimaged at will.
Ghost failed.

This bug is tracking "how to clone".  Right now that means pulling in an outside contractor to build some MDT infrastructure.
Assignee: armenzg → server-ops-releng
Component: Release Engineering → Server Operations: RelEng
QA Contact: release → zandr
Summary: recreate win64-ix-ref → figure out how to clone win64-ix-ref
(In reply to comment #6)
> mv-moz2-linux-ix-slave22 is in 650castro, is idle and can be reimaged at
> will.

This host is back in production now, so please don't reimage it.  If you need another re-image target, let me know.  Once the iX systems are back from repair and re-racked, it should be easier to find a spare system to play with.
(In reply to comment #8)
> Once the iX systems are back
> from repair and re-racked, it should be easier to find a spare system to
> play with.

Hm, I wasn't planning on re-racking anything in MV when it came back, but rather racking them up in scl1.

Should we move win64-ix-ref to scl1 sooner rather than later?
Blocks: 645024
No longer depends on: 645024
(In reply to comment #9)
> (In reply to comment #8)
> 
> Should we move win64-ix-ref to scl1 sooner rather than later?

Whenever you want.
IP:   10.250.49.173
IPMI: 10.250.49.224

RDP as Admin worked for me.
Backup Exec was suggested, but as that requires both a server installed and an agent installed on the machine to be copied, I don't this is a reasonable solution for this application, nor is it a good use of time between now and when we have someone on board to set up MDT.
win64-ix-ref is ready to be cloned from bug 661566.
Assignee: server-ops-releng → mlarrain
woot!
Depends on: 664830
win64-ix-ref is at scl1 now, working on implementing WDS and MDT for deploying images.
was able to build out a new ref build(without the tools so far) and able to capture that image. Bug submitted for ACL change so I can finish installing the mozbuild tools and recapture again.
Which bug is that?
Trying to run 'patch -p0 < /e/setup/b128.patch and getting an error'

can't find file to patch at input line 12

I also tried running it as 'patch -p1 < /e/setup/b128.patch'

After talking with dustin was able to get it to work with command

'patch -p2 < /e/setup/b128.patch'

Will update the wiki.
I finished building the ref image and was success at capturing the image to WDS/MDT. Will now have the image tested to verify it operates as expected.
I created the MDT/WDS task sequence to push the image I just need a second slave to attempt the install to. currently slave-20 has the working image on it. Will consult in the morning with spec. ops team on which w64 slave I can verify with.
These machines:

w64-ix-slave10
w64-ix-slave12
w64-ix-slave17
w64-ix-slave19
w64-ix-slave20
w64-ix-slave21
w64-ix-slave22
w64-ix-slave23
w64-ix-slave24

are on your network and all are fair game.
Give a man his own personal game preserve, and the first thing he wants to do is hunt off-premises..
(In reply to comment #20)
> I finished building the ref image and was success at capturing the image to
> WDS/MDT. Will now have the image tested to verify it operates as expected.

/me dances ;) woot!

Let me know when you I can put them on preproduction.
Need a script to be able to run the automation code that deals with the D and E drive. MDT/WDS doesn't copy over other partitions other then the Primary C:\ drive. I can have MDT/WDS install MozillaBuild, Visual Studio 2010, DirectX SDK, and Windows 7 SDK to the locations they need to be in.I Code that I need run in a script would be;

wget http://www.tortall.net/projects/yasm/releases/vsyasm-1.1.0-win64.zip
unzip vsyasm-1.1.0-win64.zip
mv vsyasm.exe /d/mozilla-build/msys/bin/yasm.exe
rm vsyasm.* readme.txt

#!/bin/sh
set -ex
ssh -i ~/.ssh/ffxbld_dsa ffxbld@aus2-staging.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@dm-symbolpush01.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage-old.mozilla.org exit
ssh -i ~/.ssh/xrbld_dsa xrbld@stage.mozilla.org exit

cd /e/setup
wget -Ob128.patch --no-check-certificate http://hg.mozilla.org/build/twisted/raw-rev/0b441f0f68b4
wget -Osubproc.patch --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=451277
cd /d/mozilla-build/python/lib/site-packages
cd twisted
patch -p0 < /e/setup/subproc.patch
patch -p2 < /e/setup/b128.patch

cd /e/setup
hg clone http://hg.mozilla.org/build/buildbot
cd buildbot
hg up -r BUILDBOT_PRODUCTION
python setup.py install

cd /c/
wget http://hg.mozilla.org/build/puppet-manifests/raw-file/d55a0757f0f0/modules/buildslave/files/runslave.py
cd /c/Users/cltbld/AppData/Roaming/Microsoft/Windows/Start\ Menu/Programs/Startup
wget -Obuildbot.bat http://hg.mozilla.org/build/opsi-package-sources/raw-file/2d51f32ff0f2/buildbot-startup/CLIENT_DATA/buildbot.bat
cd /d/mozilla-build
wget -Ostart-buildbot.bat https://bugzilla.mozilla.org/attachment.cgi?id=538014

mkdir /c/tmp
mkdir -p /e/builds/moz2_slave
Side note: The applications need to be in .msi format though.
(In reply to comment #25)
> #!/bin/sh
> set -ex
> ssh -i ~/.ssh/ffxbld_dsa ffxbld@aus2-staging.mozilla.org exit
> ssh -i ~/.ssh/ffxbld_dsa ffxbld@dm-symbolpush01.mozilla.org exit
> ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org exit
> ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage-old.mozilla.org exit
> ssh -i ~/.ssh/xrbld_dsa xrbld@stage.mozilla.org exit
This is not needed to be setup. This is just a verification step to verify that the production keys are on the machine.
It is more for documenting than anything else.
Ignore the last two I am just editing my CustomSettings.ini file to capture all the partitions. (Yay for people finding loopholes! in Microsoft deploy code)
Going to speak with Armen in the morning to discuss a few options.
Contacted Microsoft this morning and working with one of there senior level techs. They are gathering documentation to verify that the task of capturing and deploying multi-partition images is possible.
MS is going to be giving me a call in the AM (hopefully not at 5am again) and we are going to work out everything revolving around the capture and deploy of multi-partition images.
Building an image for Armen that will combine the C and D drive I will type up a walk through on how I build the image out over the weekend.
Built out the new ref machine for Armen to test. There is no longer a D:\ drive. I have moved all the D:\ drive tools to C:\Tool. The E:\ drive is still in tact. Armen will either need to use IPMI or will need special access to VPN so he can RDP into the machine to test it. Zandr when you get in can you work with Armen to get him access to http://w64-ix-slave21-mgmt.build.scl1.mozilla.com
I apologize it is C:\tools
Depends on: 667547
I will be setting the new machine up on bug 667547.
I am having trouble downloading the ISOs that I need.
I filed bug 667600.
I have added bug notes accordingly.
Trying now to connect w64-ix-slave21 (the slave without D partition) to staging master.

Filed bug 668104 to deal with it.
Will be proceeding with a different approuch due to Microsoft taking to long to get back to me. I will be writing a bash script to run hopefully during the deploy process for the E:\ drive. that will entail; 
#! /bin/bash
#
# Part 1
#
mkdir /e/setup
cd /e/setup
wget -Ob128.patch --no-check-certificate http://hg.mozilla.org/build/twisted/raw-rev/0b441f0f68b4
wget -Osubproc.patch --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=451277
cd /d/mozilla-build/python/lib/site-packages
cd twisted
patch -p0 < /e/setup/subproc.patch
patch -p2 < /e/setup/subproc.patch
#
# Part 2
#
cd /e/setup
hg clone http://hg.mozilla.org/build/buildbot
cd buildbot
hg up -r BUILDBOT_PRODUCTION
python setup.py install
digipengi the only thing I need on the slave is to create the folder:
E:\builds\moz2_slave
and to make the drive not indexable.

Everything else making reference to that hard disk was just to put thing somewhere. I have adjusted the reference platform notes to exclude mentioning the E drive and assume that I install everything without it.

For instance, once buildbot is checked out I run python setup.py install which will install buildbot under C:\mozilla-build\python\Lib\site-packages.
At that point we can dump buildbot's source code.

Makes sense?
Depends on: 668617
We fixed bug 668617 to don't even need to create E:\builds\moz2_slave by IT but instead by runslave.py.

The only thing we need to check for the E drive is to have indexation disabled.
I pulled the updated runslave.py down to the machine and renamed the old one runslave.py.old will have armen check it before I delete and capture the image.
(In reply to comment #42)
> I pulled the updated runslave.py down to the machine and renamed the old one
> runslave.py.old will have armen check it before I delete and capture the
> image.

It is kosher now.

All yours to clone.
Who wants good news? I do! I have cloned and deployed successfully. I will now leave you all the weekend to embrace the win.
Excellent news! I am so glad we have accomplished this milestone :)

Party time :D
Everything seems to be set with cloning now. I can finally close this ticket.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Just for the record, after few slaves that were cloned we figured out an auto-login issue which we are debugging in bug 645024.

All status updates are in bug 645024.
FTR, I was able to make a clonezilla image of a win2k8 VM with C/D/E drives and restore a working copy into another VM.  I'm guessing the problem is only with clones onto physical hardware?
I would not be able to say.
Even with MDT (Microsoft Deployment Toolkit) we were not able to clone it having more than one drive.
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.