1055794 - Use runner on Windows

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Start runner at startup, and then run Buildbot from it instead of from a scheduled task.

Comment 2

[Morgan Phillips [:mrrrgn]]

Attachment: runner windows buildbot task

This task will start buildbot on windows when run under cygwin via:

python -c runner.cfg tasks.d

Comment 3

[Morgan Phillips [:mrrrgn]]

Attachment: https://github.com/mrrrgn/build-runner/tree/windoze

This branch can be used in its entirety to start runner on a Windows host. All of the changes made here can be accomplished via puppet templates, since only paths are being changed.

Comment 4

[Morgan Phillips [:mrrrgn]]

Edit * You can launch buildbot just fine without using cygwin/bash.exe

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Morgan Phillips [:mrrrgn] from comment #4)
> Edit * You can launch buildbot just fine without using cygwin/bash.exe

This is scary!  I vaguely recall that the effects of not running with bash were subtle -- some particular job that relied on PATH or some other environment variable.  I wonder if we shouldn't try to change this separately, just to minimize the moving pieces.

The new buildbot.py seems to share a lot in common with modules/runner/templates/tasks/buildbot.py.erb -- could those be combined?

Comment 6

[Morgan Phillips [:mrrrgn]]

(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> (In reply to Morgan Phillips [:mrrrgn] from comment #4)
> > Edit * You can launch buildbot just fine without using cygwin/bash.exe
> 
> This is scary!  I vaguely recall that the effects of not running with bash
> were subtle -- some particular job that relied on PATH or some other
> environment variable.  I wonder if we shouldn't try to change this
> separately, just to minimize the moving pieces.
> 
> The new buildbot.py seems to share a lot in common with
> modules/runner/templates/tasks/buildbot.py.erb -- could those be combined?

I reused the runner template exactly, in fact, if we can use puppet with the windows hosts it will only require stubbing out template variables (same as the recent OSX changes).

In the task itself I start runner via bash.exe from the subprocess command itself so I think starting runner via bash.exe as well is redundant; especially since I'm using the '--login' argument which stubs out environment variables. https://github.com/mrrrgn/build-runner/blob/windoze/tasks.d/4-buildbot.py#L16

I'm using the '--login' argument because I noticed it in the current startup batch script btw.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, that makes sense then -- indeed, double-bashing probably isn't necessary!

Comment 8

[Morgan Phillips [:mrrrgn]]

Attachment: runner-windows.zip

Comment 9

[Morgan Phillips [:mrrrgn]]

Could you help me get this deployed?

Comment 10

[Mark Cornmesser [:markco]]

For sure. What all are we looking to deploy? runner-windows.zip? And schedule task to start runner?

Comment 11

[Morgan Phillips [:mrrrgn]]

(In reply to Mark Cornmesser [:markco] from comment #10)
> For sure. What all are we looking to deploy? runner-windows.zip? And
> schedule task to start runner?

Yay, yes, basically we should replace start-buildbot.bat (or whatever script is starting buildbot) with something that starts runner. To start runner, we just want to cd into its directory and use python. I've been using a batch script that just does:

cd c:\mozilla-build\runner
python runner.py -c runner.cfg tasks.d

I'll attach the batch script for review.

Comment 12

[Morgan Phillips [:mrrrgn]]

Attachment: start-runner.bat

This assumes that runner-windows.zip has been unpacked to c:\mozilla-build\runner

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Let's not dump more stuff in mozilla-build :)

On POSIX, runner ends up in /opt/runner.  I'd be game to create C:\opt to replicate that, or put it in C:\tools (a la bug 1097349) and maybe even move runner to /tools on POSIX.

Also, mrrrgn, please check in a procedure for creating the ZIP file - even if we won't be using that particular technique in the puppety future.

Comment 14

[Morgan Phillips [:mrrrgn]]

(In reply to Dustin J. Mitchell [:dustin] from comment #13)
> Let's not dump more stuff in mozilla-build :)
> 
> On POSIX, runner ends up in /opt/runner.  I'd be game to create C:\opt to
> replicate that, or put it in C:\tools (a la bug 1097349) and maybe even move
> runner to /tools on POSIX.
> 
> Also, mrrrgn, please check in a procedure for creating the ZIP file - even
> if we won't be using that particular technique in the puppety future.

Good point, and very fair. For the time being I'll create a branch that we can grab from GitHub come to think of it. github.com/mozilla/build-runner/tree/windows

Comment 15

[Morgan Phillips [:mrrrgn]]

Attachment: start-runner.bat

I'm not sure if it's acceptable to allow mkdir/git clone to fail; but I'm going with it initially here. Especially as this is expected to be a temporary measure.

Comment 16

[Morgan Phillips [:mrrrgn]]

Comment on attachment 8524239 [details]
start-runner.bat

Eh, actually, going to try for something a smidge better here. This is kind of lame even for a temporary fix.

Comment 17

[Morgan Phillips [:mrrrgn]]

Attachment: Set interpreter explicitly in runner.

I had coerced runner to work on windows via shell=True in Popen, then forgot about how brittle that was. In short, setting shell=True somehow allows us to take advantage of the nice environment set up by cygwin's bash and get around the fact that cmd.exe can't deal with hashbangs natively.

That sort of dependency could lead to lots of confusion if it ever breaks down. These changes make the interpreter being used for each task explicit. We can use this on windows to make things behave nice in a way that's simple to debug.

Comment 18

[Mark Cornmesser [:markco]]

Is cygwin going to be a prerequisite for runner?

Comment 19

[Morgan Phillips [:mrrrgn]]

(In reply to Mark Cornmesser [:markco] from comment #18)
> Is cygwin going to be a prerequisite for runner?

It is. It runs buildbot via c:\mozilla-build\msys\bin\bash.exe

Comment 20

[Justin Wood (:Callek)]

(In reply to Morgan Phillips [:mrrrgn] from comment #19)
> (In reply to Mark Cornmesser [:markco] from comment #18)
> > Is cygwin going to be a prerequisite for runner?
> 
> It is. It runs buildbot via c:\mozilla-build\msys\bin\bash.exe

to be clear, this is not cygwin, its MSYS. Cygwin has a whole host of conflicting things with MSYS and some complications with keeping both around.

Comment 21

[Morgan Phillips [:mrrrgn]]

Attachment: runner-windows.zip

This is the latest runner taken from the runner windows repo: https://github.com/mozilla/build-runner-windows/archive/0.0.0.zip

Comment 22

[Morgan Phillips [:mrrrgn]]

Attachment: start-runner.bat

Here I'm assuming that c:\opt\runner will exist and be populated by the archive at: https://github.com/mozilla/build-runner-windows/archive/0.0.0.zip

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 8524299 [details] [review]
Set interpreter explicitly in runner.

It was merged

Comment 24

[Morgan Phillips [:mrrrgn]]

(In reply to Justin Wood (:Callek) from comment #20)
> (In reply to Morgan Phillips [:mrrrgn] from comment #19)
> > (In reply to Mark Cornmesser [:markco] from comment #18)
> > > Is cygwin going to be a prerequisite for runner?
> > 
> > It is. It runs buildbot via c:\mozilla-build\msys\bin\bash.exe
> 
> to be clear, this is not cygwin, its MSYS. Cygwin has a whole host of
> conflicting things with MSYS and some complications with keeping both around.

I was confused, I see. I went with c:\mozilla-build\msys\bin\bash.exe because I noticed that's how buildbot is currently being started.

Comment 25

[Morgan Phillips [:mrrrgn]]

I'm noticing compilation errors on my test machine. I looked at one in production and noticed that it had this environment variable set: MAKE_MODE=unix

I believe this difference is causing the problem. http://dev-master1.srv.releng.scl3.mozilla.com:8046/builders/WINNT%205.2%20mozilla-central%20leak%20test%20non-unified/builds/89/steps/compile/logs/stdio

Of course, the real root of the issue is, I don't know how these environment variables are being set. We'll need to delay rolling out windows runner until I figure this out.

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

I don't have any particular advice -- I know env variables are a little different on Windows, and I'm sure MSYS bash and Python are doing some interesting things to make the world look POSIXy, too.  I am happy to see that the errors aren't subtle, though -- it looks like when you hit on the fix it will be pretty clear that it works.  And hopefully you'll have a good idea of *why* it works, too -- then we can have less cargo to cult around in the future :)

Comment 27

[Morgan Phillips [:mrrrgn]]

Attachment: start-runner.bat

Comment 28

[Morgan Phillips [:mrrrgn]]

Attachment: start-runner-win64.bat

Comment 29

[Morgan Phillips [:mrrrgn]]

I'm trying not to cargo cult anything; but it's still a bit opaque to me how all of parts of the environment are working together. That said, these startup scripts are working with runner, as placed in c:\opt\runner

Comment 30

[Armen [:armenzg]]

There are some environment variables added directly to the Windows machines through the imaging process.
Other through the buildbot start up scripts (msys/.bat files), others through b-c and others through mozharness (not sure if applies in here at all). I'm not saying that I agree with any of it :)

If you paste the log I can have a look.

I have look at releng repos, mozilla-build and trunk and I can't find where MAKE_MODE comes from.
Does GPO install anything funky?

Comment 31

[Morgan Phillips [:mrrrgn]]

This is working in my tests with the following steps btw:

mkdir c:\opt
cd c:\opt
wget https://github.com/mozilla/build-runner-windows/archive/0.0.0.zip
unzip 0.0.0
mv build-runner-windows-0.0.0 runner

then use the start-runner.bat or start-runner-win64.bat scripts.

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8525481 [details]
start-runner.bat

The environment, shell, and MSYS details hidden in these patches remain mysterious to me, but I don't see anything I don't like.

Comment 33

[Morgan Phillips [:mrrrgn]]

Attachment: runner.windows.diff

The only thing I'm not clear on is how to create a startup service. I only copied start-builbot.xml to try to make this happen. Is there more to it?

Comment 34

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8544231 [details] [diff] [review]
runner.windows.diff

Review of attachment 8544231 [details] [diff] [review]:
-----------------------------------------------------------------

Minor bits from me, but I'd like Mark's feedback too.

::: modules/buildslave/manifests/startup/runner.pp
@@ +43,5 @@
> +                'c:/programdata/puppetagain/start-buildbot.xml':
> +                    ensure => absent;
> +            }
> +
> +        }

These probably aren't necesary - when we deploy this in prod, we'll be reimaging machines from the current non-puppet status.

::: modules/runner/manifests/init.pp
@@ +8,5 @@
>      include runner::service
>      include runner::settings
>      include packages::mozilla::python27
>  
> +    $interpreter = $runner::settings::interpreter

Might want to move this closer to the template usage below, just for clarity (I wondered for a while why the python::virtualenv parameter 'python' wasn't set to $interpreter)

::: modules/runner/manifests/service.pp
@@ +67,5 @@
> +            include dirs::programdata::puppetagain
> +
> +            $builder_username = $users::builder::username
> +            $puppet_semaphore = 'C:\ProgramData\PuppetAgain\puppetcomplete.semaphore'
> +            # Batch file to start buildbot

s/buildbot/runner/g

@@ +88,5 @@
> +            shared::execonce { "startrunner":
> +                command => "$schtasks /Create /XML c:\\programdata\\puppetagain\\start-runner.xml /tn StartRunner",
> +                require => File['c:/programdata/puppetagain/start-runner.xml'],
> +            }
> +            service {

I don't think runner ends up being a service here - instead, it's a scheduled task.  Mark can confirm..

Comment 35

[Mark Cornmesser [:markco]]

Comment on attachment 8544231 [details] [diff] [review]
runner.windows.diff

Review of attachment 8544231 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/runner/manifests/service.pp
@@ +88,5 @@
> +            shared::execonce { "startrunner":
> +                command => "$schtasks /Create /XML c:\\programdata\\puppetagain\\start-runner.xml /tn StartRunner",
> +                require => File['c:/programdata/puppetagain/start-runner.xml'],
> +            }
> +            service {

It doesn't appear that runner would be a service here. Is there is something in the runner.exe that installs and starts it as a service? 
IS the, "service {"runner":" throwing an error or warning?

Comment 36

[Morgan Phillips [:mrrrgn]]

(In reply to Mark Cornmesser [:markco] from comment #35)
> Comment on attachment 8544231 [details] [diff] [review]
> runner.windows.diff
> 
> Review of attachment 8544231 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: modules/runner/manifests/service.pp
> @@ +88,5 @@
> > +            shared::execonce { "startrunner":
> > +                command => "$schtasks /Create /XML c:\\programdata\\puppetagain\\start-runner.xml /tn StartRunner",
> > +                require => File['c:/programdata/puppetagain/start-runner.xml'],
> > +            }
> > +            service {
> 
> It doesn't appear that runner would be a service here. Is there is something
> in the runner.exe that installs and starts it as a service? 
> IS the, "service {"runner":" throwing an error or warning?

The service here does not throw a warning. Rather, not having it caused an error thanks to some other steps having a requirement for a runner "service" existing before they will execute. I added the service to avoid putting in tons of conditionals.

Comment 37

[Dustin J. Mitchell [:dustin] (he/him)]

That a Service resource for an undefined service doesn't error out sounds like a puppet bug, which could be patched at any time.  That would be a nasty surprise down the road.

Rather than conditionals, you could define a $runner_service_requirements = [] on Windows and = [Service['runner']] elsewhere.

Comment 38

[Morgan Phillips [:mrrrgn]]

Attachment: runner.windows.diff

This addresses the issues up above. Seems to install properly on my test box.

Comment 39

[Morgan Phillips [:mrrrgn]]

Attachment: runner.windows.diff

Comment 40

[Morgan Phillips [:mrrrgn]]

Attachment: runner.windows.diff

Sorry for the new patches, there was a template problem which only affected osx puppet clients. Fixed now.

Comment 41

[Rail Aliiev [:rail]]

Comment on attachment 8551423 [details] [diff] [review]
runner.windows.diff

Review of attachment 8551423 [details] [diff] [review]:
-----------------------------------------------------------------

stamping the xmls and bat files. Otherwise looks straight forward.

Comment 42

[Morgan Phillips [:mrrrgn]]

I don't mean to add more load; but is there any way we can go ahead and deploy these changes under our current model (non-puppet)? That will allow us to track the status of our windows hosts via the runner dashboard, which comes in handy for seeing problems pop up.

I can create a zip archive with the compiled bat/xml files if necessary. Runner is already available in our pip repository on releg-puppet2.srv.releng.scl3.mozilla.com

Comment 43

[Justin Wood (:Callek)]

(In reply to Morgan Phillips [:mrrrgn] from comment #42)
> I don't mean to add more load; but is there any way we can go ahead and
> deploy these changes under our current model (non-puppet)? That will allow
> us to track the status of our windows hosts via the runner dashboard, which
> comes in handy for seeing problems pop up.
> 
> I can create a zip archive with the compiled bat/xml files if necessary.
> Runner is already available in our pip repository on
> releg-puppet2.srv.releng.scl3.mozilla.com

fwiw, I support this, *or* I support puppet on windows turned on without runner. I don't support a split in what one does vs the other.

E.g. if we have runner in GPO we should have runner in puppet, if we don't we shouldn't. Having a divergence here will incur much more cost in terms of turning on puppet-on-windows (imho) since if there are issues we won't know what caused them easily, and have a harder time triaging/fixing them.

Comment 44

[Morgan Phillips [:mrrrgn]]

I backed out the changes for now, I'll sit on the patch until windows-puppet deploys or we can get runner going via GPO.

Comment 45

[Mark Cornmesser [:markco]]

(In reply to Morgan Phillips [:mrrrgn] from comment #42)
> I don't mean to add more load; but is there any way we can go ahead and
> deploy these changes under our current model (non-puppet)? That will allow
> us to track the status of our windows hosts via the runner dashboard, which
> comes in handy for seeing problems pop up.
> 
> I can create a zip archive with the compiled bat/xml files if necessary.
> Runner is already available in our pip repository on
> releg-puppet2.srv.releng.scl3.mozilla.com

I am down to do this, but let's get Amy's opinion/OK or not OK to divert time to it.

Comment 46

[Amy Rich [:arr] [:arich]]

Talked with Morgan and Mark, and this sounds like a great use of a day or two to get us a bunch of reporting. Proceed, please!

Comment 47

[Morgan Phillips [:mrrrgn]]

Attachment: opt.tar.gz

This has everything you should need to get runner working. After you extract it from c:\ try running c:\opt\start-runner.bat and that should start buildbot.

Comment 48

[Morgan Phillips [:mrrrgn]]

Attachment: opt-1.8.tar.gz

This version of runner is compatible with python 2.6 > and comes with a new script install-runner.bat which will put runner into the buildbot virtual environment. The start-runner.bat script has also been modified in light of this.

This should work on any windows host by just un-tarring to /c/
then running /c/opt/install-runner.bat and /c/opt/start-runner.bat

Comment 49

[Morgan Phillips [:mrrrgn]]

Attachment: opt-1.8-fixed-halt.tar.gz

Windows is failing on halt.py because of the hashbang, this changes the halt script to NT batch. It also fixes a bug in the start-runner.bat script.

Comment 50

[Mark Cornmesser [:markco]]

A GPO is ready to push runner out to the domain machines. 

How do we want to proceed?

Comment 51

[Morgan Phillips [:mrrrgn]]

(In reply to Mark Cornmesser [:markco] from comment #50)
> A GPO is ready to push runner out to the domain machines. 
> 
> How do we want to proceed?

Sweet! I saw the data from your test make a nice blip on the runner dashboards.

Can we roll it out on a subset of the machines first? The first issue I can imagine is some interference between the old buildbot startup and the runner buildbot startup. 

In any case, I'm ready for this as soon as possible.

Comment 52

[Mark Cornmesser [:markco]]

I going to need info Jordan on this. 

Jordan, is there a few 2008 machines we could roll runner out to? Even though we are in the begin steps of transitioning 2008 to Puppet managed, I think this is worthwhile since the time of the transition is largely unknown.

Comment 53

[Jordan Lund (:jlund)]

(In reply to Mark Cornmesser [:markco] from comment #52)
>
> Jordan, is there a few 2008 machines we could roll runner out to? .

sure. feel free to do 1 at a time or whatever works for you with say:
b-2008-ix-0100 -> b-2008-ix-0109

the above gives you 10 to eventually get up to and they are numerically beside each other. I checked each one out. It's hard to find a slave that has a wall of solid green build as I think win64 and some specific builders are not the most reliable. At any rate, the above do not seem any worse than the rest :)

Comment 54

[Mark Cornmesser [:markco]]

Thanks Jordan!

I have disabled 0100 through 0104 to start. If the current builds finish in a timely manner I will move them into the testing OU tonight. If not I will move them in the morning.

Comment 55

[Mark Cornmesser [:markco]]

0100 through 0104 have been moved to the testing OU, force a group policy update, and rebooted each machine.

They are ready to be re-enabled.

Comment 56

[Phil Ringnalda (:philor)]

They got reenabled, then I just disabled them, since they burn the compile step with "which: python2.7: unknown command".

Comment 57

[Morgan Phillips [:mrrrgn]]

I modified b-2008-ix-0102 and put it back into play for the night to see if the situation improves. Sadly, the Windows path situation is black magic to me, but I used a working start-buildbot.bat from a normal slave and copied it over the buildbot task in runner. 

Keeping an eye on it, and will disable again if need be.

Comment 58

[Phil Ringnalda (:philor)]

Please can we test this in some other way, like say with a dev master running non-production jobs?

Either 0104 got reenabled too, or I missed disabling it, and so it burned l10n nightlies all night long, and because of the way we report them (to nowhere, basically), instead of being retriggered and wasting several hours of developer and sheriff time like the rest of the jobs we're burning, those just got thrown on the floor, and too bad if anyone wanted to test those locales today.

Comment 59

[Mark Cornmesser [:markco]]

I am going to go ahead and re-image those machines. They should be ready to go back in production within a the couple hours.

Comment 60

[Jordan Lund (:jlund)]

(In reply to Mark Cornmesser [:markco] from comment #59)
> I am going to go ahead and re-image those machines. They should be ready to
> go back in production within a the couple hours.

IIUC - we are re-imaging them to pre-runner. so they are fine to all enable at once when ready. thanks markco :)

Comment 61

[Morgan Phillips [:mrrrgn]]

So, runner is handling jobs here: http://dev-master2.bb.releng.use1.mozilla.com:8046/buildslaves/ix-mn-w0864-001 it looks like there was a problem with the environment when we tried to deploy with GPO. I've solved the Makefile issue by adding this to the runner config:

[env]
MAKE_MODE=unix
MSYSTEM=MINGW32

However, I'm not convinced we won't run into a path issue. The good news is, it's simple to transplant the correct environment variables (from a working machine) to the runner config. Once we can deploy with puppet, this _should_ be trivial to get right (we manually set the path for some of our OSX machines in their runner config already for example).

I'll continue running builds on this slave, and any other I can get my hands on to look for troubles.

Comment 62

[Morgan Phillips [:mrrrgn]]

Attachment: opt.tar.gz

This has everything fixed up, and a bulk clobber task.

Comment 63

[Morgan Phillips [:mrrrgn]]

bulk clobbering works => https://pastebin.mozilla.org/8833439 and jobs are completing successfully on our test machine: http://buildbot-master87.bb.releng.scl3.mozilla.com:8101/buildslaves/b-2008-ix-0058 I think we're ready to roll this baby out.

Comment 64

[Morgan Phillips [:mrrrgn]]

Two problems with the latest deploy:

1.) The PATH set by runner is messed up because of problems with slashes (this is the joy of Windows paths)
"PATH=C:\mozilla-buildpython27;C:\mozilla-buildbuildbotve\scripts; .... " It should be
"PATH=C:\mozilla-build\python27;C:\mozilla-build\buildbotve\scripts; ...."

2.) Runner and the old system are running side by side. One of the boxes was in the process of running two jobs at the same time, last I checked. B-2008-IX-0057

Comment 65

[Morgan Phillips [:mrrrgn]]

We can tell when a job has been launched by runner because it will have the "RUNNER_CONFIG_CMD" environment variable set. See: http://buildbot-master87.bb.releng.scl3.mozilla.com:8101/builders/WINNT%206.1%20x86-64%20try%20spidermonkey_try-plain%20build/builds/60/steps/run_script/logs/stdio

Comment 66

[Morgan Phillips [:mrrrgn]]

I can create a new config file that attempts to mitigate this slashing issue.

Comment 67

[Morgan Phillips [:mrrrgn]]

Attachment: opt.tar.gz

Here opt/runner/runner.cfg has been modified with a path that starts with Python. This is to mitigate an issue of the system's default python being used. The system python breaks jobs because it's 2.6

Comment 68

[Rob Thijssen [:grenade (EET/UTC+0300)]]

I've updated the runner gpo files with the modified opt. I don't know what's going on with the 2 buildbot processes on 0057. The runner test pool is 0050 to 0054 so (I think) that one shouldn't be running the new runner stuff.

Comment 69

[Rob Thijssen [:grenade (EET/UTC+0300)]]

cancelling fb req...

Comment 70

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8608879 [details]
opt.tar.gz

deployed

Comment 71

[Morgan Phillips [:mrrrgn]]

Looking at the jobs running on the 0050 to 0054 machines, I'm not seeing the RUNNER_CONFIG_CMD environment variable set (which is always set by runner), so it looks like the buildbot here was still started using the old script. :(

Comment 72

[Morgan Phillips [:mrrrgn]]

Attachment: opt.tar.gz

The windows runner was trying to run every task with python: breaking batch scripts.

Comment 73

[Morgan Phillips [:mrrrgn]]

Installed the new tarball on B-2008-IX-0057 with successful clobber and bb initialization.

Comment 74

[Rob Thijssen [:grenade (EET/UTC+0300)]]

New opt installed via GPO on:
 - B-2008-IX-0050
 - B-2008-IX-0051
 - B-2008-IX-0052
 - B-2008-IX-0053
 - B-2008-IX-0054

Comment 75

[Morgan Phillips [:mrrrgn]]

2015-05-28 06:06:27,078 - try_w32-d_sm-plaindebug-000000:Server is forcing a clobber, init
There's a clobberer bug. Have we seen this before on Windows?

iated by [e-mail]
2015-05-28 06:06:27,078 - try_w32-d_sm-plaindebug-000000:Clobbering...
2015-05-28 06:06:27,078 - Skipping buildprops.json
2015-05-28 06:06:27,079 - Skipping last-clobber
2015-05-28 06:06:27,079 - Skipping scripts
2015-05-28 06:06:27,079 - Removing src.deleteme/
c:/opt/clobberer.py:163: DeprecationWarning: BaseException.message has been deprecated as
of Python 2.6
  log.error("[clobber failed]: %s", e.message)
2015-05-28 06:06:27,082 - [clobber failed]:

Comment 76

[Morgan Phillips [:mrrrgn]]

Attachment: opt.tar.gz

Comment 77

[Morgan Phillips [:mrrrgn]]

Attachment: opt.tar.gz

Fixes a string encoding error that the clobberer client had (windows filesystem encoding was not utf-8 compatible) and introduces better logging.

Comment 78

[Q]

Working to deploy new opt.tar.gz to b-208-ix-0050 - 054

Comment 79

[Q]

The wmi filters and item level targets for gpo need some review before this gets pushed but it will be done today.

Comment 80

[Q]

The gpo logic is now correct and the install runs only installs once. I checked 054 and 052 and they look like they are getting the right config and runner is starting

Comment 81

[Q]

expanding runner gpo to encompass b-2008-ix-0020 - 0059

Comment 82

[Q]

per conversations with mrrrgn the expanded try roll-out looks good we are going to let this run over the weekend and chat about expanding to all try machines during the work week.

Comment 83

[Morgan Phillips [:mrrrgn]]

Looking to roll out runner to all try machines today.

Comment 84

[Q]

All try machines confirmed. Per conversations at whistler b-2008-ix-0171 is using runner. Next step is add 0171 - 0173 as the rest if 017x are in try using runner already.

Q

Comment 85

[Q]

The content of the current wmi filter used to include machines in the runner deployment process:

select * from Win32_ComputerSystem where (Name LIKE "B-2008-IX-005%" OR Name LIKE "B-2008-IX-004%" OR Name LIKE "B-2008-IX-003%" OR Name LIKE "B-2008-IX-002%" OR NAME = "B-2008-IX-0060" OR NAME = "B-2008-IX-0061" OR NAME = "B-2008-IX-0062" OR NAME = "B-2008-IX-0063" OR NAME = "B-2008-IX-0064" OR NAME = "B-2008-IX-0170" OR NAME = "B-2008-IX-0173" OR NAME = "B-2008-IX-0174" OR NAME = "B-2008-IX-0175" OR NAME = "B-2008-IX-0176" OR NAME = "B-2008-IX-0177" OR NAME = "B-2008-IX-0178" OR NAME = "B-2008-IX-0179" OR NAME = "B-2008-IX-0180" OR NAME = "B-2008-IX-0181" OR NAME = "B-2008-IX-0182" OR NAME = "B-2008-IX-0183" OR NAME = "B-2008-IX-0184" )

And to exclude runner machines from having the buildbot schedule task reinstalled:

select * from Win32_ComputerSystem where NOT  (Name LIKE "B-2008-IX-005%" OR Name LIKE "B-2008-IX-004%" OR Name LIKE "B-2008-IX-003%" OR Name LIKE "B-2008-IX-002%" OR NAME = "B-2008-IX-0060" OR NAME = "B-2008-IX-0061" OR NAME = "B-2008-IX-0062" OR NAME = "B-2008-IX-0063" OR NAME = "B-2008-IX-0064" OR NAME = "B-2008-IX-0170" OR NAME = "B-2008-IX-0173" OR NAME = "B-2008-IX-0174" OR NAME = "B-2008-IX-0175" OR NAME = "B-2008-IX-0176" OR NAME = "B-2008-IX-0177" OR NAME = "B-2008-IX-0178" OR NAME = "B-2008-IX-0179" OR NAME = "B-2008-IX-0180" OR NAME = "B-2008-IX-0181" OR NAME = "B-2008-IX-0182" OR NAME = "B-2008-IX-0183" OR NAME = "B-2008-IX-0184" )

Comment 86

[Q]

Adding B-2008-IX-017x to the mix this only adds 171 - 173 to the build runner testing pool as other machines are already in try or testing for Puppett.

Comment 87

[Q]

Per IRC discussions added the rest of 018x and 006x for the sake of filter cleanliness and tracking:

select * from Win32_ComputerSystem where (Name LIKE "B-2008-IX-018%" OR Name LIKE "B-2008-IX-017%" OR NAME LIKE "B-2008-IX-006%" OR Name LIKE "B-2008-IX-005%" OR Name LIKE "B-2008-IX-004%" OR Name LIKE "B-2008-IX-003%" OR Name LIKE "B-2008-IX-002%")

select * from Win32_ComputerSystem where NOT (Name LIKE "B-2008-IX-018%" OR Name LIKE "B-2008-IX-017%" OR NAME LIKE "B-2008-IX-006%" OR Name LIKE "B-2008-IX-005%" OR Name LIKE "B-2008-IX-004%" OR Name LIKE "B-2008-IX-003%" OR Name LIKE "B-2008-IX-002%")

Comment 88

[Q]

Expanding to include b-2008-ix-000* b-2008-ix-001* b-2008-ix-016* That adds 27 machines using runner