1134980 - run a puppetized slave in try

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mark Cornmesser [:markco]]

b-2008-ix-0010 has the current Windows builder Puppet configuration applied to it. The next steps, I think< would be trying to run some test builds against and see how it fares.

Comment 1

[Jordan Lund (:jlund)]

roger. adding to my todo on Monday.

Comment 2

[Jordan Lund (:jlund)]

I didn't get to this today but it is on my priority for tomorrow. Will sync up then.

Comment 3

[Jordan Lund (:jlund)]

the first job lasted for two whole seconds! let the fun begin! :D

error in RunProcess._startCommand
Traceback (most recent call last):
  File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\lib\site-packages\buildslave\runprocess.py", line 377, in start
    self._startCommand()
  File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\lib\site-packages\buildslave\runprocess.py", line 497, in _startCommand
    usePTY=self.usePTY)
  File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\lib\site-packages\buildslave\runprocess.py", line 523, in _spawnProcess
    path, usePTY=usePTY)
  File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\lib\site-packages\twisted\internet\posixbase.py", line 335, in spawnProcess
    raise NotImplementedError, "spawnProcess not available since pywin32 is not installed."
NotImplementedError: spawnProcess not available since pywin32 is not installed.

remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last):
Failure: buildslave.exceptions.AbandonChain: -1
]




so I think we want to be using /c/mozilla-build/python/python.exe looks like this is using /c/mozilla-build/buildbot-0.8.4-pre-moz6/Scripts/python.exe which may be the same thing. Either way, it appears like python doesn't have many modules installed:

cltbld@B-2008-IX-0010 ~
$ ls /c/mozilla-build/python/Lib/site-packages/
README.txt                   easy-install.pth             setuptools-0.6c11-py2.7.egg  setuptools.pth

what we normally expect:
cltbld@B-2008-IX-0003 /c/mozilla-build/python
$ ls /c/mozilla-build/python/lib/site-packages/
PyWin32.chm                          easy-install.pth                     pywin32-214-py2.6.egg-info           twisted/
README.txt                           isapi/                               pywin32.pth                          win32/
Twisted-9.0.0-py2.6.egg-info         pythoncom.py                         pywin32.version.txt                  win32com/
adodbapi/                            pythoncom.pyc                        pywin32_system32/                    win32comext/
axscript/                            pythoncom.pyo                        setuptools-0.6c11-py2.6.egg          zope/
buildbot-0.8.0-py2.6.egg/            pythonwin/                           setuptools.pth                       zope.interface-3.5.2-py2.6.egg-info


notice win32 is there for B-2008-IX-0003

markco: I can be around tomorrow during the day and try to do some fast cycles on this with you if you like.

Comment 4

[Mark Cornmesser [:markco]]

Hmm... fun. This will take me a little bit of time to go through. From the look of things, on the domain machines those modules were included in a mass file copy. With Puppet this will most likely turn into several package installs.

Comment 5

[Mark Cornmesser [:markco]]

The patch for pywin has just landed. Whenever you are ready to give it another try just reboot, and it should pick up the change.

Comment 6

[Mark Cornmesser [:markco]]

Jordan, When do you think you could give this another shot?

Comment 7

[Jordan Lund (:jlund)]

I triggered a build last night. looking at log now shows we yielded the same results.

I think this is because our job is using c:\mozilla-build\buildbot-0.8.4-pre-moz6\Scripts\python.exe. I think what we want is it to use c:\mozilla-build\python\python.exe

I'll come find you on irc and see if we can iterate a bit quicker on this.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

So pywin32 is installed in the "system" python, while Buildbot's running in the virtualenv and doesn't see those packages.  The easiest fix will be to build the virtualenv with --system-site-packages, so that we can accesss pywin32.  The harder fix will involve installing pywin32 in the virtualenvs :(

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1134980.patch

Comment 10

[Mark Cornmesser [:markco]]

Attachment: BUG1134980-package.patch

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8570729 [details] [diff] [review]
BUG1134980-package.patch

Review of attachment 8570729 [details] [diff] [review]:
-----------------------------------------------------------------

The technique looks good, but two problems:
 1. different version (216 instead of 218).  You should be able to download the correct package from sourceforge.
 2. this class should include and require => Class['packages::mozilla::python27'] so that it's installed already

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8570674 [details] [diff] [review]
bug1134980.patch

I'll land this on Monday -- remind me if necessary!

Comment 13

[Mark Cornmesser [:markco]]

Attachment: BUG1134980-package.patch

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8571456 [details] [diff] [review]
BUG1134980-package.patch

Review of attachment 8571456 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good with these two points addressed.

::: modules/packages/manifests/pywin32.pp
@@ +7,5 @@
> +    include packages::mozilla::python27
> +
> +    exec { "pywin32":
> +        require => Class['packages::mozilla::python27'],
> +        command => 'C:\mozilla-build\python\Scripts\easy_install.exe  http://puppetagain.pub.build.mozilla.org/data/repos/EXEs/pywin32-218.win32-py2.7.exe',

..http://${::config::data_server}/... (just like in pkgmsi, pkgzip, etc.)

@@ +8,5 @@
> +
> +    exec { "pywin32":
> +        require => Class['packages::mozilla::python27'],
> +        command => 'C:\mozilla-build\python\Scripts\easy_install.exe  http://puppetagain.pub.build.mozilla.org/data/repos/EXEs/pywin32-218.win32-py2.7.exe',
> +        creates => 'C:\mozilla-build\\Lib\\site-packages\\pywin32-218-py2.7-win32.egg';

Are the double-backslashes required here?  The first backslash isn't doubled.

Comment 15

[Mark Cornmesser [:markco]]

Comment on attachment 8571456 [details] [diff] [review]
BUG1134980-package.patch

Review of attachment 8571456 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/08678064aec8

Per irc callek said he will merge this production later today.

Comment 16

[Mark Cornmesser [:markco]]

Hey Jordan could you give this another try, please?

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8570674 [details] [diff] [review]
bug1134980.patch

remote:   https://hg.mozilla.org/build/puppet/rev/69e2e5605ca0

Comment 18

[Jordan Lund (:jlund)]

(In reply to Mark Cornmesser [:markco] from comment #16)
> Hey Jordan could you give this another try, please?

sure thing, I'll add this to my queue today

Comment 19

[Jordan Lund (:jlund)]

progress

we made it past that step now so obviously it picked up pywin32 \o/

we are now failing here:
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/47/steps/clone_scripts/logs/stdio

snippet:
'hg' 'clone' 'https://hg.mozilla.org/build/mozharness' 'scripts'
abort: could not find web.cacerts: C:\stage\mozilla-build\hg\hgrc.d\cacert.pem
program finished with exit code 255
elapsedTime=1.508000

full job:
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/47

markco: if you get unblocked from this error and want to retrigger, I've left my master for you hooked up to this slave so all you need to do is go to this url:
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build

and then click on the button that says 'force build'

one caveat is this certain failure causes a RETRY so if you see that it keeps trying to run this over and over, you will have click on the button that says 'stop build'. side note: this is bad and we probably don't want to retry for errors like these.

Comment 20

[Mark Cornmesser [:markco]]

Quick note; what is causing this in C:\mozilla-build\hg\hgrc.d\Paths.rc. In which it points cacerts to C:\stage... directory.

Comment 21

[Mark Cornmesser [:markco]]

Attachment: BUG1134980-hg_paths_rc.patch

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8573393 [details] [diff] [review]
BUG1134980-hg_paths_rc.patch

Review of attachment 8573393 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me for a quick fix.  Ted, is this the right permanent solution, or is this something that mozbuild should eventually fix?

Comment 23

[(not currently active) Ted Mielczarek]

Comment on attachment 8573393 [details] [diff] [review]
BUG1134980-hg_paths_rc.patch

RyanVM has taken over ownership of MozillaBuild, so I'm going to defer to him.

Comment 24

[Mark Cornmesser [:markco]]

triggered another build: 
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13632/steps/reboot/logs/stdio

  WIX_351728_PATH=c:/mozilla-build/wix-351728
  _=c:\mozilla-build\buildbot-0.8.4-pre-moz6\Scripts\python
 using PTY: False
The system cannot find the path specified.
program finished with exit code 1
elapsedTime=0.109000

Comment 25

[Dustin J. Mitchell [:dustin] (he/him)]

I *think* that error comes from Buildbot being unable to find the executable given in the command.  It's not visible in the bit pasted above, but in this case it's

'c:/mozilla-build/python27/python' '-u' 'scripts/external_tools/count_and_reboot.py' '-f' '../reboot_count.txt' '-n' '1' '-z'
 in dir c:\builds\moz2_slave\m-cen-w32-00000000000000000000\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['c:/mozilla-build/python27/python', '-u', 'scripts/external_tools/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']
 environment:

and C:\mozilla-build\python27\python.exe does exist.  I'd wonder if this is a slash issue, but this command has been working fine for years.. so I don't really have any ideas :(

Comment 26

[Chris AtLee [:catlee]]

\ vs / ?

Comment 27

[Mark Cornmesser [:markco]]

The python27 directory does not exist in the puppet configuration. It is not included in the MozillaBuild package. It ended up on domain machines due to  https://bugzilla.mozilla.org/show_bug.cgi?id=987535.

Comment 28

[Dustin J. Mitchell [:dustin] (he/him)]

K, so we need to reproduce that in puppet, right?  Eep, should be fun.

Comment 29

[Mark Cornmesser [:markco]]

Attachment: BUG1134980-python27.patch

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8573985 [details] [diff] [review]
BUG1134980-python27.patch

Review of attachment 8573985 [details] [diff] [review]:
-----------------------------------------------------------------

Assuming c:\mozilla-build\python is python2.7, this looks good.  Is this what we do on GPO-controlled machines?

Comment 31

[Mark Cornmesser [:markco]]

 
> Assuming c:\mozilla-build\python is python2.7, this looks good.  Is this
> what we do on GPO-controlled machines?

On the GPO machines we had multiple versions of python at some point, 2.5, 2.6, and 2.7. Which what lead to coping and renaming of Python to Python 2.7. I figured a link would be better than copying and renaming, and the easiest way to modify it in the future.

Comment 32

[Mark Cornmesser [:markco]]

After the python27 link was in place, I triggered another build. Which actually ran for about 26 minutes but failed out. 

http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13633/steps/run_script/logs/stdio

digging through the log now.

Comment 33

[Mark Cornmesser [:markco]]

08:13:46     INFO -  configure:22883: checking for Unicode NSIS version 2.46 or greater
08:13:46     INFO -  configure: error: To build the installer you must have the latest MozillaBuild or Unicode NSIS version 2.46 or greater in your path.

Comment 34

[Jordan Lund (:jlund)]

(In reply to Mark Cornmesser [:markco] from comment #33)
> 08:13:46     INFO -  configure:22883: checking for Unicode NSIS version 2.46
> or greater
> 08:13:46     INFO -  configure: error: To build the installer you must have
> the latest MozillaBuild or Unicode NSIS version 2.46 or greater in your path.

that exe comes from: http://mxr.mozilla.org/build/source/mozharness/configs/builds/releng_base_windows_32_builds.py#78

which I can see your build is using:
08:12:35     INFO -  'PATH': 'C:/mozilla-build/nsis-3.0a2;C:/mozilla-build/nsis-2.46u;C:/mozilla-build/python27;C:/mozilla-build/buildbotve/scripts;etc...


not sure where things are falling apart yet

running it locally also looks good
cltbld@B-2008-IX-0010 /c/mozilla-build/nsis-2.46u
$ /c/mozilla-build/nsis-2.46u/makensisu-2.46.exe -version
v2.46-Unicode

Comment 35

[Mark Cornmesser [:markco]]

Attachment: BUG1134980-nsis.patch

It seems that renaming the nsis-3.0a2 allows buildbot to use the older version.

Comment 36

[Mark Cornmesser [:markco]]

After renaming the nsis-3.0a2, the build seemed to have picked up 2.46u and continued farther. Here is the last log: 

http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13634/steps/run_script/logs/stdio

Comment 37

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8574127 [details] [diff] [review]
BUG1134980-nsis.patch

Review of attachment 8574127 [details] [diff] [review]:
-----------------------------------------------------------------

Yuck, nice work figuring that out, and including a comment in the source here.

Comment 38

[Jordan Lund (:jlund)]

oh right, your puppet version has both and it was taking over the 2.46 version. I see what you mean now.

Comment 39

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8573393 [details] [diff] [review]
BUG1134980-hg_paths_rc.patch

This sounds like bug 679808, which was fixed in Mozillabuild back in 2011. IIRC, aren't our production machines using some ancient hacked-up copy of Mozillabuild at this point?

Comment 40

[Dustin J. Mitchell [:dustin] (he/him)]

It says we're using "Latest" so that can't possibly be out of date, right?  Mark, when things quiet down we should probably make a point of adjusting the manifests to use a mozilla_build ZIP with a version number in it :)

Comment 41

[Mark Cornmesser [:markco]]

Jlund, when you have moment could you take a look at this log: 
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13634/steps/run_script/logs/stdio

I am having a bit of difficulty figuring out what is causing the failure.

Comment 42

[Jordan Lund (:jlund)]

(In reply to Mark Cornmesser [:markco] from comment #41)
> Jlund, when you have moment could you take a look at this log: 
> http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.
> 2%20mozilla-central%20build/builds/13634/steps/run_script/logs/stdio
> 
> I am having a bit of difficulty figuring out what is causing the failure.

for some reason my comment did not save last night. I dug around a bit and saw:

11:42:25     INFO -  Usage: signtool.py [options] file [file ...]
11:42:25     INFO -  If no include patterns are specified, all files will be considered. -i/-x only
11:42:25     INFO -  have effect when signing entire directories.
11:42:25     INFO -  signtool.py: error: no hosts capable of signing formats: emevoucher
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/toolkit/mozapps/installer/packager.mk:77: recipe for target 'make-package-internal' failed
11:42:25     INFO -  mozmake.EXE[4]: *** [make-package-internal] Error 2
11:42:25     INFO -  mozmake.EXE[4]: Leaving directory 'c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/obj-firefox/browser/installer'
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/toolkit/mozapps/installer/packager.mk:81: recipe for target 'make-package' failed
11:42:25     INFO -  mozmake.EXE[3]: *** [make-package] Error 2
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/config/rules.mk:541: recipe for target 'default' failed
11:42:25     INFO -  mozmake.EXE[2]: *** [default] Error 2
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/browser/build.mk:9: recipe for target 'package' failed
11:42:25     INFO -  mozmake.EXE[1]: *** [package] Error 2
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/build/moz-automation.mk:128: recipe for target 'automation/package' failed
11:42:25     INFO -  mozmake.EXE: *** [automation/package] Error 2

I think the latter errors was a domino effect from signing failing with:
11:42:25     INFO -  signtool.py: error: no hosts capable of signing formats: emevoucher

Turns out we needed a new format added to our master. the creds/formatting for signing lives in:
/builds/buildbot/jlund/build-master/master/passwords.py

I updated that file to include emevoucher, reconfigured the master (make reconfig), and re-spun the failing job last night: http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13635

looks like the job is a lot healthier now \o/

Comment 43

[Mark Cornmesser [:markco]]

Comment on attachment 8573393 [details] [diff] [review]
BUG1134980-hg_paths_rc.patch

Review of attachment 8573393 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/33c06b57df88

Comment 44

[Mark Cornmesser [:markco]]

Comment on attachment 8573985 [details] [diff] [review]
BUG1134980-python27.patch

Review of attachment 8573985 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/7d2ec1254654

Comment 45

[Mark Cornmesser [:markco]]

Comment on attachment 8573985 [details] [diff] [review]
BUG1134980-python27.patch

Review of attachment 8573985 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/1a529e95cbb3

Comment 46

[Mark Cornmesser [:markco]]

Comment on attachment 8574127 [details] [diff] [review]
BUG1134980-nsis.patch

Review of attachment 8574127 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/f7a790106065

Comment 47

[Dustin J. Mitchell [:dustin] (he/him)]

remote:   https://hg.mozilla.org/build/puppet/rev/edc5902fc40b

Comment 48

[Mark Cornmesser [:markco]]

Attachment: BUG1134980-nsis-correction.patch

Comment 49

[Mark Cornmesser [:markco]]

Comment on attachment 8577267 [details] [diff] [review]
BUG1134980-nsis-correction.patch

 https://hg.mozilla.org/build/puppet/rev/0bd2c271b526

Comment 50

[Mark Cornmesser [:markco]]

Attachment: BUG1134980-py27link.patch

Comment 51

[Mark Cornmesser [:markco]]

Comment on attachment 8578730 [details] [diff] [review]
BUG1134980-py27link.patch

The link attribute tries to relink on subsequent runs causing an error. Replaced it with an exec.

Comment 52

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8578730 [details] [diff] [review]
BUG1134980-py27link.patch

Review of attachment 8578730 [details] [diff] [review]:
-----------------------------------------------------------------

This is kind of yucky, but it will go away with a new mozilla-build (or its replacement).  So just two little issues here and good to commit.

::: modules/packages/manifests/mozilla/mozilla_build.pp
@@ +20,5 @@
>      # Currently Buildbot looks for python27 on Windows slaves.
>      # This will need to be revisited when Python is updated in the Mozillabuild package.
> +    # Exec is being used here since the file link attribute throws errors on subsequent runs on Windows
> +    exec {
> +        "pyhton_27_link":

Spelling :)

@@ -28,5 @@
>      # When buildbot looks for nsis, version 3.0a2 causes an error "Error initalizing CEXEBuild: can't open file: integer expression expected"
>      # Reference https://bugzilla.mozilla.org/show_bug.cgi?id=989531
>      # Temporary work around is to move version 3.0a2, so that 2.46u is used  
>      exec {"move_nsis_3_0a2":
> -        command => 'C:\mozilla-build\msys\bin\mv.exe  C:\mozilla-build\nsis-3.0a2 C:\mozilla-build\HOLDnsis-3.0a2 /y',

Why drop the /y here?

Comment 53

[Jordan Lund (:jlund)]

the above issues were resolved and last staging build was successful. we are going to enable this machine in try to see if it is successful there too

Comment 54

[Jordan Lund (:jlund)]

fyi - I am on PTO until May 13th.

these slaves are currently disabled