683976 - Add buildbot and updated runslave.py to the w64 image

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task, P5)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Bug 679818 suggests that we're not using reference machines anymore, so the work done in bug 637347 needs to be incorporated into the install process for future w64 systems.

I'm happy to help, but I don't know how that's done.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

To be clear, this is about the *new* version of Buildbot - 0.8.4-pre-moz2.

Comment 2

[Matthew Larrain[:MaRu]]

I am adding custom run scripts for the image to update to the latest buildbot and runslave.py now.

Comment 3

[Matthew Larrain[:MaRu]]

Added a custom task during deployment to run the buildbot batch file that dustin sent me. For now I have added the batch file to the image but am playing on having the task have each new deploy look to a share on the WDS server for the batch file. This will make it easier to edit the batch file and keep our standards of trying to get away from having "golden images" I still need to test a push of a new image to verify the batch file works.

Comment 4

[Armen [:armenzg]]

Let me know if you need a host to try on and I will pull one or few for you.

Comment 5

[Matthew Larrain[:MaRu]]

I was able to get this updated and I just need a host to test the new install on. Armen let me know which one I can work with please and thank you.

Comment 6

[Armen [:armenzg]]

(In reply to Matthew Larrain[:digipengi] from comment #5)
> I was able to get this updated and I just need a host to test the new
> install on. Armen let me know which one I can work with please and thank you.

You can use w64-ix-slave41. Sorry for the delay.

Comment 7

[Matthew Larrain[:MaRu]]

Testing image update on w64-ix-slave41

Comment 8

[Armen [:armenzg]]

As digipengi mentions he is having trouble deploying runslave.py to the w64 machines as placing it under "C:\" requires dealing with an UAC prompt (I never learn how to circumvent those).

The reason this was placed in "C:\" was because it was common denominator for win2008, win2003, win7 and winxp.

If we decide to go down the path of placing it somewhere where we won't need to deal with admin problems we need to take the following into consideration:

1) buildbot.bat (aka buildbot-win64.bat) calls "C:\mozilla-build\start-buildbot.bat" [1]
2) start-buildbot.bat (aka start-buildbot-win64.bat) calls "python /c/runslave.py" [2]

This means that we would have to patch start-buildbot-win64.bat and re-deploy it.
This also implies that we would have to make all w64 slaves to get patched as well (runslave.py at a newer location and re-deploy start-buildbot.bat to point to the new location of runslave.py).

This is all document on the ref docs [3].

Let me know if this makes sense or not :).

[1] http://mxr.mozilla.org/build/source/puppet-manifests/modules/buildslave/files/buildbot-win64.bat
[2] http://hg.mozilla.org/build/puppet-manifests/file/c6e02dee3589/modules/buildslave/files/start-buildbot-win64.bat#l109
[3] https://wiki.mozilla.org/ReferencePlatforms/Win64#Startup_scripts

Comment 9

[Matthew Larrain[:MaRu]]

I am able to have WDS/MDT deploy the batch file but there is a part in it that requires user interaction of typing in the admin password.

Comment 10

[Armen [:armenzg]]

I will work on this and get digipengi a script that won't need admin privileges.

This will be a Q1 goal.

Comment 11

[Armen [:armenzg]]

I have too much in my plate and have to prioritize work.

This is not a pressing issue as Windows 64-bit has not become a tier-1 platform.

There is also debate if we should use group policy up to certain point and from there we puppetize the rest with Windows (this bug sounds like a good candidate).

Comment 12

[Chris Cooper [:coop] (he/him)]

Unless we end up building win32 on win64, this bug is very low priority.

Comment 13

[Phil Ringnalda (:philor)]

I downtimed the "w64-ix-slave41.winbuild.scl1:PING is CRITICAL: PING CRITICAL - Packet loss = 100%" alert for 180 days, if this lets that slave loose before then you'll want to undo that.

Comment 14

[Aki Sasaki (not active)]

Digipengi: could you send this bug back or file a new one about putting w64-ix-slave41 back in the pool once you're done? I'm going to remove w64-ix-slave41 from bug 684019 so it stops lingering.

Comment 15

[Armen [:armenzg]]

MaRu, I will try to install tomorrow morning buildbot assuming the newer location of runslave.py.

Steps to fix the runslave.py issue:
1) wget -OC:\Tools\runslave.py http://hg.mozilla.org/build/puppet-manifests/raw-file/tip/modules/buildslave/files/runslave.py
2) modify and deploy newer C:\mozilla-build\start-buildbot.bat

diff --git a/modules/buildslave/files/start-buildbot-win64.bat b/modules/buildslave/files/start-buildbot-win64.bat
--- a/modules/buildslave/files/start-buildbot-win64.bat
+++ b/modules/buildslave/files/start-buildbot-win64.bat
@@ -101,11 +101,11 @@ if "%WIN64%" == "1" (
 cd "%USERPROFILE%"
 :start
 
 echo "%date% %time% - About to run runslave.py"
 
 REM running this via 'bash' is critical - bash adds a bunch of items to PATH
 REM which the build steps expect to find.
 
-"%MOZILLABUILD%\msys\bin\bash" --login -c "python /c/runslave.py"
+"%MOZILLABUILD%\msys\bin\bash" --login -c "python /c/Tools/runslave.py"
 
 echo "%date% %time% - runslave.py finished"

Comment 16

[Armen [:armenzg]]

Attachment: [puppet-manifests] C:\Tools\runslave.py instead of C:\runslave.py

This removes the need to be administrator to deploy runslave.py
This is working for me on staging.

Comment 17

[Armen [:armenzg]]

Attachment: [puppet-manifests] install buildbot-0.8.4 to win64 machines and deploy runslave.py to C:\Tools

The only changes I have made to the original is:
* change production-puppet to scl3-production-puppet
* remove the echo off
* C drive for mozilla-build instead of D drive
* wget runslave.py to C:\Tools instead of C:\
** no more need for runas administrator

Comment 18

[Armen [:armenzg]]

I need to figure out which way to go.

Here is how things are called:
buildbot.bat -> C:\mozilla-build\start-buildbot.bat -> C:\runslave.py

To deploy the newer buildbot we need also a newer runslave.py.
To deploy runslave.py unto C:\ we need to run as admin which does not allow us to automate it.  We need to fix this so we can use GPO.

We can deploy runslave.py to C:\Tools without any trouble *but* that means we need to deploy a start-buildbot.bat that calls C:\Tools\runslave.py instead of C:\runslave.py.
Unfortunately, start-builtbot.bat is owned by admin.
MaRu, can this be fixed? I guess we can do it manually. That would suck but it will allow us to move on.

On another note, we could deploy a newer buildbot.bat that calls to C:\**Tools**\start-buildbot.bat (rather than mozilla-buildbot).

At the end of the road we should not:
* have mozilla-build owned by Administrator
* delete C:\runslave.py

Comment 19

[Amy Rich [:arr] [:arich]]

Armen: did you mean scl-production-puppet?  That's the puppet master in the datacenter where the w64 machines are (scl1).

Comment 20

[Armen [:armenzg]]

Attachment: [puppet-manifests] C:\Tools is the new magic

This patch puts runslave.py and start-buildbot.bat in C:\Tools.
This patch puts a new buildbot.bat in place that calls C:\Tools\start-buildbot.bat

Current:
buildbot.bat -> C:\mozilla-build\start-buildbot.bat -> C:\runslave.py

Proposed change:
buildbot.bat -> C:\Tools\start-buildbot.bat -> C:\Tools\runslave.py

Works for you?

Comment 21

[Armen [:armenzg]]

Note: you can ignore pointing to my people account and think of URLs to puppet-manifests.

Comment 22

[Armen [:armenzg]]

Comment on attachment 626894 [details] [diff] [review]
[puppet-manifests] C:\Tools is the new magic

This doesn't work.
guess-msvc.bat is under C:\mozilla-build (which is called by start-buildbot.bat).

Comment 23

[Armen [:armenzg]]

Attachment: update-buildbot-win64.bat

This is what MaRu is now testing.

Comment 24

[Armen [:armenzg]]

Attachment: update-buildbot-win64.bat (no "runas")

I emailed this last night to MaRu.
I think we don't need the "runas" command if we are running it as Administrator. Otherwise, it will prompt us regardless of which user we run this script as.

This script only differs in the last command compared to the previous one.

Comment 25

[Armen [:armenzg]]

MaRu fixed this.

The only manual step now is fixing the auto-login (bug 758769).