figure out how to clone win64-ix-ref

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps
P3
normal
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: armenzg, Assigned: MaRu)

Tracking

Details

(Reporter)

Description

7 years ago
zandr had to re-image the machine in bug 645024.

If I recreate the machine we can get a consultant he has lined up to try to clone it with other different tool than CloneZilla.

The IP seems to be 10.250.49.173 and we have to figure out the IPMI address and if RDP is enabled.
(Reporter)

Updated

7 years ago
Depends on: 661015

Comment 2

7 years ago
Host is up @ 10.250.49.239

Comment 3

7 years ago
Host is running through a ghost image to an attached USB drive.  Once this finishes, we'll try to reimage.

Comment 4

7 years ago
image backup done.  need a machine i can try to restore to.

ghost iso is @ \\fs2\IT\Microsoft\NGH1501_AllWin_English_SrdAndProd.iso .
From talking with mrz; this bug is tracking making a copy of a vanilla install of win2008 (64bit). 

The machine being copied *from* does not have any of Armen's work on it.

The purpose of this experiment is only to see if IT have tools that can take a working image from one win2008 machine, and put it onto another blank machine which boots correctly.
Depends on: 645024
(In reply to comment #4)
> image backup done.  need a machine i can try to restore to.
> 
> ghost iso is @ \\fs2\IT\Microsoft\NGH1501_AllWin_English_SrdAndProd.iso .

mv-moz2-linux-ix-slave22 is in 650castro, is idle and can be reimaged at will.

Comment 7

7 years ago
Ghost failed.

This bug is tracking "how to clone".  Right now that means pulling in an outside contractor to build some MDT infrastructure.
Assignee: armenzg → server-ops-releng
Component: Release Engineering → Server Operations: RelEng
QA Contact: release → zandr
Summary: recreate win64-ix-ref → figure out how to clone win64-ix-ref
(In reply to comment #6)
> mv-moz2-linux-ix-slave22 is in 650castro, is idle and can be reimaged at
> will.

This host is back in production now, so please don't reimage it.  If you need another re-image target, let me know.  Once the iX systems are back from repair and re-racked, it should be easier to find a spare system to play with.
(In reply to comment #8)
> Once the iX systems are back
> from repair and re-racked, it should be easier to find a spare system to
> play with.

Hm, I wasn't planning on re-racking anything in MV when it came back, but rather racking them up in scl1.

Should we move win64-ix-ref to scl1 sooner rather than later?
(Reporter)

Updated

7 years ago
Blocks: 645024
No longer depends on: 645024
(Reporter)

Comment 10

7 years ago
(In reply to comment #9)
> (In reply to comment #8)
> 
> Should we move win64-ix-ref to scl1 sooner rather than later?

Whenever you want.
(Reporter)

Comment 11

7 years ago
IP:   10.250.49.173
IPMI: 10.250.49.224

RDP as Admin worked for me.
Backup Exec was suggested, but as that requires both a server installed and an agent installed on the machine to be copied, I don't this is a reasonable solution for this application, nor is it a good use of time between now and when we have someone on board to set up MDT.
(Reporter)

Comment 13

7 years ago
win64-ix-ref is ready to be cloned from bug 661566.
Assignee: server-ops-releng → mlarrain
(Reporter)

Comment 14

7 years ago
woot!
(Reporter)

Updated

7 years ago
Depends on: 664830
(Assignee)

Comment 15

7 years ago
win64-ix-ref is at scl1 now, working on implementing WDS and MDT for deploying images.
(Assignee)

Comment 16

7 years ago
was able to build out a new ref build(without the tools so far) and able to capture that image. Bug submitted for ACL change so I can finish installing the mozbuild tools and recapture again.
Which bug is that?
(Assignee)

Comment 18

7 years ago
Bug 665782
(Assignee)

Comment 19

7 years ago
Trying to run 'patch -p0 < /e/setup/b128.patch and getting an error'

can't find file to patch at input line 12

I also tried running it as 'patch -p1 < /e/setup/b128.patch'

After talking with dustin was able to get it to work with command

'patch -p2 < /e/setup/b128.patch'

Will update the wiki.
(Assignee)

Comment 20

7 years ago
I finished building the ref image and was success at capturing the image to WDS/MDT. Will now have the image tested to verify it operates as expected.
(Assignee)

Comment 21

7 years ago
I created the MDT/WDS task sequence to push the image I just need a second slave to attempt the install to. currently slave-20 has the working image on it. Will consult in the morning with spec. ops team on which w64 slave I can verify with.
These machines:

w64-ix-slave10
w64-ix-slave12
w64-ix-slave17
w64-ix-slave19
w64-ix-slave20
w64-ix-slave21
w64-ix-slave22
w64-ix-slave23
w64-ix-slave24

are on your network and all are fair game.
Give a man his own personal game preserve, and the first thing he wants to do is hunt off-premises..
(Reporter)

Comment 24

7 years ago
(In reply to comment #20)
> I finished building the ref image and was success at capturing the image to
> WDS/MDT. Will now have the image tested to verify it operates as expected.

/me dances ;) woot!

Let me know when you I can put them on preproduction.
(Assignee)

Comment 25

7 years ago
Need a script to be able to run the automation code that deals with the D and E drive. MDT/WDS doesn't copy over other partitions other then the Primary C:\ drive. I can have MDT/WDS install MozillaBuild, Visual Studio 2010, DirectX SDK, and Windows 7 SDK to the locations they need to be in.I Code that I need run in a script would be;

wget http://www.tortall.net/projects/yasm/releases/vsyasm-1.1.0-win64.zip
unzip vsyasm-1.1.0-win64.zip
mv vsyasm.exe /d/mozilla-build/msys/bin/yasm.exe
rm vsyasm.* readme.txt

#!/bin/sh
set -ex
ssh -i ~/.ssh/ffxbld_dsa ffxbld@aus2-staging.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@dm-symbolpush01.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org exit
ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage-old.mozilla.org exit
ssh -i ~/.ssh/xrbld_dsa xrbld@stage.mozilla.org exit

cd /e/setup
wget -Ob128.patch --no-check-certificate http://hg.mozilla.org/build/twisted/raw-rev/0b441f0f68b4
wget -Osubproc.patch --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=451277
cd /d/mozilla-build/python/lib/site-packages
cd twisted
patch -p0 < /e/setup/subproc.patch
patch -p2 < /e/setup/b128.patch

cd /e/setup
hg clone http://hg.mozilla.org/build/buildbot
cd buildbot
hg up -r BUILDBOT_PRODUCTION
python setup.py install

cd /c/
wget http://hg.mozilla.org/build/puppet-manifests/raw-file/d55a0757f0f0/modules/buildslave/files/runslave.py
cd /c/Users/cltbld/AppData/Roaming/Microsoft/Windows/Start\ Menu/Programs/Startup
wget -Obuildbot.bat http://hg.mozilla.org/build/opsi-package-sources/raw-file/2d51f32ff0f2/buildbot-startup/CLIENT_DATA/buildbot.bat
cd /d/mozilla-build
wget -Ostart-buildbot.bat https://bugzilla.mozilla.org/attachment.cgi?id=538014

mkdir /c/tmp
mkdir -p /e/builds/moz2_slave
(Assignee)

Comment 26

7 years ago
Side note: The applications need to be in .msi format though.
(Reporter)

Comment 27

7 years ago
(In reply to comment #25)
> #!/bin/sh
> set -ex
> ssh -i ~/.ssh/ffxbld_dsa ffxbld@aus2-staging.mozilla.org exit
> ssh -i ~/.ssh/ffxbld_dsa ffxbld@dm-symbolpush01.mozilla.org exit
> ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org exit
> ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage-old.mozilla.org exit
> ssh -i ~/.ssh/xrbld_dsa xrbld@stage.mozilla.org exit
This is not needed to be setup. This is just a verification step to verify that the production keys are on the machine.
It is more for documenting than anything else.
(Assignee)

Comment 28

7 years ago
Ignore the last two I am just editing my CustomSettings.ini file to capture all the partitions. (Yay for people finding loopholes! in Microsoft deploy code)
(Assignee)

Comment 29

7 years ago
Going to speak with Armen in the morning to discuss a few options.
(Assignee)

Comment 30

7 years ago
Contacted Microsoft this morning and working with one of there senior level techs. They are gathering documentation to verify that the task of capturing and deploying multi-partition images is possible.
(Assignee)

Comment 31

7 years ago
MS is going to be giving me a call in the AM (hopefully not at 5am again) and we are going to work out everything revolving around the capture and deploy of multi-partition images.
(Assignee)

Comment 32

7 years ago
Building an image for Armen that will combine the C and D drive I will type up a walk through on how I build the image out over the weekend.
(Assignee)

Comment 33

7 years ago
Built out the new ref machine for Armen to test. There is no longer a D:\ drive. I have moved all the D:\ drive tools to C:\Tool. The E:\ drive is still in tact. Armen will either need to use IPMI or will need special access to VPN so he can RDP into the machine to test it. Zandr when you get in can you work with Armen to get him access to http://w64-ix-slave21-mgmt.build.scl1.mozilla.com
(Assignee)

Comment 34

7 years ago
I apologize it is C:\tools
(Reporter)

Updated

7 years ago
Depends on: 667547
(Reporter)

Comment 35

7 years ago
I will be setting the new machine up on bug 667547.
(Reporter)

Comment 36

7 years ago
I am having trouble downloading the ISOs that I need.
I filed bug 667600.
(Assignee)

Comment 37

7 years ago
I have added bug notes accordingly.
(Reporter)

Comment 38

7 years ago
Trying now to connect w64-ix-slave21 (the slave without D partition) to staging master.

Filed bug 668104 to deal with it.
(Assignee)

Comment 39

7 years ago
Will be proceeding with a different approuch due to Microsoft taking to long to get back to me. I will be writing a bash script to run hopefully during the deploy process for the E:\ drive. that will entail; 
#! /bin/bash
#
# Part 1
#
mkdir /e/setup
cd /e/setup
wget -Ob128.patch --no-check-certificate http://hg.mozilla.org/build/twisted/raw-rev/0b441f0f68b4
wget -Osubproc.patch --no-check-certificate https://bugzilla.mozilla.org/attachment.cgi?id=451277
cd /d/mozilla-build/python/lib/site-packages
cd twisted
patch -p0 < /e/setup/subproc.patch
patch -p2 < /e/setup/subproc.patch
#
# Part 2
#
cd /e/setup
hg clone http://hg.mozilla.org/build/buildbot
cd buildbot
hg up -r BUILDBOT_PRODUCTION
python setup.py install
(Reporter)

Comment 40

7 years ago
digipengi the only thing I need on the slave is to create the folder:
E:\builds\moz2_slave
and to make the drive not indexable.

Everything else making reference to that hard disk was just to put thing somewhere. I have adjusted the reference platform notes to exclude mentioning the E drive and assume that I install everything without it.

For instance, once buildbot is checked out I run python setup.py install which will install buildbot under C:\mozilla-build\python\Lib\site-packages.
At that point we can dump buildbot's source code.

Makes sense?
(Reporter)

Updated

7 years ago
Depends on: 668617
(Reporter)

Comment 41

7 years ago
We fixed bug 668617 to don't even need to create E:\builds\moz2_slave by IT but instead by runslave.py.

The only thing we need to check for the E drive is to have indexation disabled.
(Assignee)

Comment 42

7 years ago
I pulled the updated runslave.py down to the machine and renamed the old one runslave.py.old will have armen check it before I delete and capture the image.
(Reporter)

Comment 43

7 years ago
(In reply to comment #42)
> I pulled the updated runslave.py down to the machine and renamed the old one
> runslave.py.old will have armen check it before I delete and capture the
> image.

It is kosher now.

All yours to clone.
(Assignee)

Comment 44

7 years ago
Who wants good news? I do! I have cloned and deployed successfully. I will now leave you all the weekend to embrace the win.
(Reporter)

Comment 45

7 years ago
Excellent news! I am so glad we have accomplished this milestone :)

Party time :D
(Assignee)

Comment 46

7 years ago
Everything seems to be set with cloning now. I can finally close this ticket.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 47

7 years ago
Just for the record, after few slaves that were cloned we figured out an auto-login issue which we are debugging in bug 645024.

All status updates are in bug 645024.
FTR, I was able to make a clonezilla image of a win2k8 VM with C/D/E drives and restore a working copy into another VM.  I'm guessing the problem is only with clones onto physical hardware?
(Reporter)

Comment 49

7 years ago
I would not be able to say.
Even with MDT (Microsoft Deployment Toolkit) we were not able to clone it having more than one drive.
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.