Closed Bug 1134980 Opened 9 years ago Closed 8 years ago

run a puppetized slave in try

Categories

(Release Engineering :: General, defect)

x86_64
Windows Server 2008
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: markco, Assigned: jlund)

References

Details

Attachments

(7 files, 1 obsolete file)

b-2008-ix-0010 has the current Windows builder Puppet configuration applied to it. The next steps, I think< would be trying to run some test builds against and see how it fares.
Assignee: nobody → jlund
roger. adding to my todo on Monday.
I didn't get to this today but it is on my priority for tomorrow. Will sync up then.
the first job lasted for two whole seconds! let the fun begin! :D

error in RunProcess._startCommand
Traceback (most recent call last):
  File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\lib\site-packages\buildslave\runprocess.py", line 377, in start
    self._startCommand()
  File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\lib\site-packages\buildslave\runprocess.py", line 497, in _startCommand
    usePTY=self.usePTY)
  File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\lib\site-packages\buildslave\runprocess.py", line 523, in _spawnProcess
    path, usePTY=usePTY)
  File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\lib\site-packages\twisted\internet\posixbase.py", line 335, in spawnProcess
    raise NotImplementedError, "spawnProcess not available since pywin32 is not installed."
NotImplementedError: spawnProcess not available since pywin32 is not installed.

remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last):
Failure: buildslave.exceptions.AbandonChain: -1
]




so I think we want to be using /c/mozilla-build/python/python.exe looks like this is using /c/mozilla-build/buildbot-0.8.4-pre-moz6/Scripts/python.exe which may be the same thing. Either way, it appears like python doesn't have many modules installed:

cltbld@B-2008-IX-0010 ~
$ ls /c/mozilla-build/python/Lib/site-packages/
README.txt                   easy-install.pth             setuptools-0.6c11-py2.7.egg  setuptools.pth

what we normally expect:
cltbld@B-2008-IX-0003 /c/mozilla-build/python
$ ls /c/mozilla-build/python/lib/site-packages/
PyWin32.chm                          easy-install.pth                     pywin32-214-py2.6.egg-info           twisted/
README.txt                           isapi/                               pywin32.pth                          win32/
Twisted-9.0.0-py2.6.egg-info         pythoncom.py                         pywin32.version.txt                  win32com/
adodbapi/                            pythoncom.pyc                        pywin32_system32/                    win32comext/
axscript/                            pythoncom.pyo                        setuptools-0.6c11-py2.6.egg          zope/
buildbot-0.8.0-py2.6.egg/            pythonwin/                           setuptools.pth                       zope.interface-3.5.2-py2.6.egg-info


notice win32 is there for B-2008-IX-0003

markco: I can be around tomorrow during the day and try to do some fast cycles on this with you if you like.
Flags: needinfo?(mcornmesser)
Hmm... fun. This will take me a little bit of time to go through. From the look of things, on the domain machines those modules were included in a mass file copy. With Puppet this will most likely turn into several package installs.
Flags: needinfo?(mcornmesser)
Depends on: 1136824
The patch for pywin has just landed. Whenever you are ready to give it another try just reboot, and it should pick up the change.
Jordan, When do you think you could give this another shot?
Flags: needinfo?(jlund)
I triggered a build last night. looking at log now shows we yielded the same results.

I think this is because our job is using c:\mozilla-build\buildbot-0.8.4-pre-moz6\Scripts\python.exe. I think what we want is it to use c:\mozilla-build\python\python.exe

I'll come find you on irc and see if we can iterate a bit quicker on this.
Flags: needinfo?(jlund)
So pywin32 is installed in the "system" python, while Buildbot's running in the virtualenv and doesn't see those packages.  The easiest fix will be to build the virtualenv with --system-site-packages, so that we can accesss pywin32.  The harder fix will involve installing pywin32 in the virtualenvs :(
Attached patch bug1134980.patchSplinter Review
Attachment #8570674 - Flags: review?(mcornmesser)
Attached patch BUG1134980-package.patch (obsolete) — Splinter Review
Attachment #8570729 - Flags: review?(dustin)
Attachment #8570674 - Flags: review?(mcornmesser) → review+
Comment on attachment 8570729 [details] [diff] [review]
BUG1134980-package.patch

Review of attachment 8570729 [details] [diff] [review]:
-----------------------------------------------------------------

The technique looks good, but two problems:
 1. different version (216 instead of 218).  You should be able to download the correct package from sourceforge.
 2. this class should include and require => Class['packages::mozilla::python27'] so that it's installed already
Attachment #8570729 - Flags: review?(dustin) → review-
Comment on attachment 8570674 [details] [diff] [review]
bug1134980.patch

I'll land this on Monday -- remind me if necessary!
Attachment #8570729 - Attachment is obsolete: true
Attachment #8571456 - Flags: review?(dustin)
Comment on attachment 8571456 [details] [diff] [review]
BUG1134980-package.patch

Review of attachment 8571456 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good with these two points addressed.

::: modules/packages/manifests/pywin32.pp
@@ +7,5 @@
> +    include packages::mozilla::python27
> +
> +    exec { "pywin32":
> +        require => Class['packages::mozilla::python27'],
> +        command => 'C:\mozilla-build\python\Scripts\easy_install.exe  http://puppetagain.pub.build.mozilla.org/data/repos/EXEs/pywin32-218.win32-py2.7.exe',

..http://${::config::data_server}/... (just like in pkgmsi, pkgzip, etc.)

@@ +8,5 @@
> +
> +    exec { "pywin32":
> +        require => Class['packages::mozilla::python27'],
> +        command => 'C:\mozilla-build\python\Scripts\easy_install.exe  http://puppetagain.pub.build.mozilla.org/data/repos/EXEs/pywin32-218.win32-py2.7.exe',
> +        creates => 'C:\mozilla-build\\Lib\\site-packages\\pywin32-218-py2.7-win32.egg';

Are the double-backslashes required here?  The first backslash isn't doubled.
Attachment #8571456 - Flags: review?(dustin) → review+
Comment on attachment 8571456 [details] [diff] [review]
BUG1134980-package.patch

Review of attachment 8571456 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/08678064aec8

Per irc callek said he will merge this production later today.
Attachment #8571456 - Flags: checked-in+
Hey Jordan could you give this another try, please?
Flags: needinfo?(jlund)
(In reply to Mark Cornmesser [:markco] from comment #16)
> Hey Jordan could you give this another try, please?

sure thing, I'll add this to my queue today
Flags: needinfo?(jlund)
progress

we made it past that step now so obviously it picked up pywin32 \o/

we are now failing here:
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/47/steps/clone_scripts/logs/stdio

snippet:
'hg' 'clone' 'https://hg.mozilla.org/build/mozharness' 'scripts'
abort: could not find web.cacerts: C:\stage\mozilla-build\hg\hgrc.d\cacert.pem
program finished with exit code 255
elapsedTime=1.508000

full job:
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/47

markco: if you get unblocked from this error and want to retrigger, I've left my master for you hooked up to this slave so all you need to do is go to this url:
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build

and then click on the button that says 'force build'

one caveat is this certain failure causes a RETRY so if you see that it keeps trying to run this over and over, you will have click on the button that says 'stop build'. side note: this is bad and we probably don't want to retry for errors like these.
Quick note; what is causing this in C:\mozilla-build\hg\hgrc.d\Paths.rc. In which it points cacerts to C:\stage... directory.
Attachment #8573393 - Flags: review?(dustin)
Comment on attachment 8573393 [details] [diff] [review]
BUG1134980-hg_paths_rc.patch

Review of attachment 8573393 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me for a quick fix.  Ted, is this the right permanent solution, or is this something that mozbuild should eventually fix?
Attachment #8573393 - Flags: review?(dustin)
Attachment #8573393 - Flags: review+
Attachment #8573393 - Flags: feedback?(ted)
Comment on attachment 8573393 [details] [diff] [review]
BUG1134980-hg_paths_rc.patch

RyanVM has taken over ownership of MozillaBuild, so I'm going to defer to him.
Attachment #8573393 - Flags: feedback?(ted) → feedback?(ryanvm)
triggered another build: 
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13632/steps/reboot/logs/stdio

  WIX_351728_PATH=c:/mozilla-build/wix-351728
  _=c:\mozilla-build\buildbot-0.8.4-pre-moz6\Scripts\python
 using PTY: False
The system cannot find the path specified.
program finished with exit code 1
elapsedTime=0.109000
I *think* that error comes from Buildbot being unable to find the executable given in the command.  It's not visible in the bit pasted above, but in this case it's

'c:/mozilla-build/python27/python' '-u' 'scripts/external_tools/count_and_reboot.py' '-f' '../reboot_count.txt' '-n' '1' '-z'
 in dir c:\builds\moz2_slave\m-cen-w32-00000000000000000000\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['c:/mozilla-build/python27/python', '-u', 'scripts/external_tools/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']
 environment:

and C:\mozilla-build\python27\python.exe does exist.  I'd wonder if this is a slash issue, but this command has been working fine for years.. so I don't really have any ideas :(
\ vs / ?
The python27 directory does not exist in the puppet configuration. It is not included in the MozillaBuild package. It ended up on domain machines due to  https://bugzilla.mozilla.org/show_bug.cgi?id=987535.
K, so we need to reproduce that in puppet, right?  Eep, should be fun.
Attachment #8573985 - Flags: review?(dustin)
Comment on attachment 8573985 [details] [diff] [review]
BUG1134980-python27.patch

Review of attachment 8573985 [details] [diff] [review]:
-----------------------------------------------------------------

Assuming c:\mozilla-build\python is python2.7, this looks good.  Is this what we do on GPO-controlled machines?
Attachment #8573985 - Flags: review?(dustin) → review+
 
> Assuming c:\mozilla-build\python is python2.7, this looks good.  Is this
> what we do on GPO-controlled machines?

On the GPO machines we had multiple versions of python at some point, 2.5, 2.6, and 2.7. Which what lead to coping and renaming of Python to Python 2.7. I figured a link would be better than copying and renaming, and the easiest way to modify it in the future.
After the python27 link was in place, I triggered another build. Which actually ran for about 26 minutes but failed out. 

http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13633/steps/run_script/logs/stdio

digging through the log now.
08:13:46     INFO -  configure:22883: checking for Unicode NSIS version 2.46 or greater
08:13:46     INFO -  configure: error: To build the installer you must have the latest MozillaBuild or Unicode NSIS version 2.46 or greater in your path.
(In reply to Mark Cornmesser [:markco] from comment #33)
> 08:13:46     INFO -  configure:22883: checking for Unicode NSIS version 2.46
> or greater
> 08:13:46     INFO -  configure: error: To build the installer you must have
> the latest MozillaBuild or Unicode NSIS version 2.46 or greater in your path.

that exe comes from: http://mxr.mozilla.org/build/source/mozharness/configs/builds/releng_base_windows_32_builds.py#78

which I can see your build is using:
08:12:35     INFO -  'PATH': 'C:/mozilla-build/nsis-3.0a2;C:/mozilla-build/nsis-2.46u;C:/mozilla-build/python27;C:/mozilla-build/buildbotve/scripts;etc...


not sure where things are falling apart yet

running it locally also looks good
cltbld@B-2008-IX-0010 /c/mozilla-build/nsis-2.46u
$ /c/mozilla-build/nsis-2.46u/makensisu-2.46.exe -version
v2.46-Unicode
It seems that renaming the nsis-3.0a2 allows buildbot to use the older version.
Attachment #8574127 - Flags: feedback?(dustin)
After renaming the nsis-3.0a2, the build seemed to have picked up 2.46u and continued farther. Here is the last log: 

http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13634/steps/run_script/logs/stdio
Comment on attachment 8574127 [details] [diff] [review]
BUG1134980-nsis.patch

Review of attachment 8574127 [details] [diff] [review]:
-----------------------------------------------------------------

Yuck, nice work figuring that out, and including a comment in the source here.
Attachment #8574127 - Flags: feedback?(dustin) → feedback+
oh right, your puppet version has both and it was taking over the 2.46 version. I see what you mean now.
Comment on attachment 8573393 [details] [diff] [review]
BUG1134980-hg_paths_rc.patch

This sounds like bug 679808, which was fixed in Mozillabuild back in 2011. IIRC, aren't our production machines using some ancient hacked-up copy of Mozillabuild at this point?
Attachment #8573393 - Flags: feedback?(ryanvm) → feedback+
It says we're using "Latest" so that can't possibly be out of date, right?  Mark, when things quiet down we should probably make a point of adjusting the manifests to use a mozilla_build ZIP with a version number in it :)
Jlund, when you have moment could you take a look at this log: 
http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13634/steps/run_script/logs/stdio

I am having a bit of difficulty figuring out what is causing the failure.
Flags: needinfo?(jlund)
(In reply to Mark Cornmesser [:markco] from comment #41)
> Jlund, when you have moment could you take a look at this log: 
> http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.
> 2%20mozilla-central%20build/builds/13634/steps/run_script/logs/stdio
> 
> I am having a bit of difficulty figuring out what is causing the failure.

for some reason my comment did not save last night. I dug around a bit and saw:

11:42:25     INFO -  Usage: signtool.py [options] file [file ...]
11:42:25     INFO -  If no include patterns are specified, all files will be considered. -i/-x only
11:42:25     INFO -  have effect when signing entire directories.
11:42:25     INFO -  signtool.py: error: no hosts capable of signing formats: emevoucher
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/toolkit/mozapps/installer/packager.mk:77: recipe for target 'make-package-internal' failed
11:42:25     INFO -  mozmake.EXE[4]: *** [make-package-internal] Error 2
11:42:25     INFO -  mozmake.EXE[4]: Leaving directory 'c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/obj-firefox/browser/installer'
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/toolkit/mozapps/installer/packager.mk:81: recipe for target 'make-package' failed
11:42:25     INFO -  mozmake.EXE[3]: *** [make-package] Error 2
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/config/rules.mk:541: recipe for target 'default' failed
11:42:25     INFO -  mozmake.EXE[2]: *** [default] Error 2
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/browser/build.mk:9: recipe for target 'package' failed
11:42:25     INFO -  mozmake.EXE[1]: *** [package] Error 2
11:42:25     INFO -  c:/builds/moz2_slave/m-cen-w32-00000000000000000000/build/src/build/moz-automation.mk:128: recipe for target 'automation/package' failed
11:42:25     INFO -  mozmake.EXE: *** [automation/package] Error 2

I think the latter errors was a domino effect from signing failing with:
11:42:25     INFO -  signtool.py: error: no hosts capable of signing formats: emevoucher

Turns out we needed a new format added to our master. the creds/formatting for signing lives in:
/builds/buildbot/jlund/build-master/master/passwords.py

I updated that file to include emevoucher, reconfigured the master (make reconfig), and re-spun the failing job last night: http://dev-master2.bb.releng.use1.mozilla.com:8037/builders/WINNT%205.2%20mozilla-central%20build/builds/13635

looks like the job is a lot healthier now \o/
Flags: needinfo?(jlund)
Attachment #8574127 - Flags: review?(dustin)
Comment on attachment 8573393 [details] [diff] [review]
BUG1134980-hg_paths_rc.patch

Review of attachment 8573393 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/33c06b57df88
Attachment #8573393 - Flags: checked-in+
Comment on attachment 8573985 [details] [diff] [review]
BUG1134980-python27.patch

Review of attachment 8573985 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/7d2ec1254654
Attachment #8573985 - Flags: checked-in+
Attachment #8574127 - Flags: review?(dustin) → review+
Attachment #8573985 - Flags: checked-in+
Comment on attachment 8573985 [details] [diff] [review]
BUG1134980-python27.patch

Review of attachment 8573985 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/1a529e95cbb3
Attachment #8573985 - Flags: checked-in+
Comment on attachment 8574127 [details] [diff] [review]
BUG1134980-nsis.patch

Review of attachment 8574127 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/f7a790106065
Attachment #8574127 - Flags: checked-in+
Attachment #8577267 - Flags: review?(dustin)
Attachment #8577267 - Flags: review?(dustin) → review+
Attachment #8578730 - Flags: review?(dustin)
Comment on attachment 8578730 [details] [diff] [review]
BUG1134980-py27link.patch

The link attribute tries to relink on subsequent runs causing an error. Replaced it with an exec.
Comment on attachment 8578730 [details] [diff] [review]
BUG1134980-py27link.patch

Review of attachment 8578730 [details] [diff] [review]:
-----------------------------------------------------------------

This is kind of yucky, but it will go away with a new mozilla-build (or its replacement).  So just two little issues here and good to commit.

::: modules/packages/manifests/mozilla/mozilla_build.pp
@@ +20,5 @@
>      # Currently Buildbot looks for python27 on Windows slaves.
>      # This will need to be revisited when Python is updated in the Mozillabuild package.
> +    # Exec is being used here since the file link attribute throws errors on subsequent runs on Windows
> +    exec {
> +        "pyhton_27_link":

Spelling :)

@@ -28,5 @@
>      # When buildbot looks for nsis, version 3.0a2 causes an error "Error initalizing CEXEBuild: can't open file: integer expression expected"
>      # Reference https://bugzilla.mozilla.org/show_bug.cgi?id=989531
>      # Temporary work around is to move version 3.0a2, so that 2.46u is used  
>      exec {"move_nsis_3_0a2":
> -        command => 'C:\mozilla-build\msys\bin\mv.exe  C:\mozilla-build\nsis-3.0a2 C:\mozilla-build\HOLDnsis-3.0a2 /y',

Why drop the /y here?
Attachment #8578730 - Flags: review?(dustin) → review+
QA Contact: pmoore → mshal
the above issues were resolved and last staging build was successful. we are going to enable this machine in try to see if it is successful there too
Summary: Hook up b-2008-ix-0010 to a dev master → Hook up b-2008-ix-0010, a puppetized slave, to a try master
Summary: Hook up b-2008-ix-0010, a puppetized slave, to a try master → run a puppetized slave in try
fyi - I am on PTO until May 13th.

these slaves are currently disabled
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: