711299 - Deploy python 2.7.3 to all test machines

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Aki Sasaki (not active)]

Analogous to bug 602908; we should probably target the same version.

I was able to find some workarounds in bug 706244, but this would have helped in getting peptest live a lot smoother+faster.

Comment 1

[Aki Sasaki (not active)]

We need to install for all users, not just this user, or we get this ugliness on Windows:

00:25:34     INFO - Running command: ['c:/mozilla-build/python25/python', 'c:/mozilla-build/buildbotve/virtualenv.py', '--no-site-packages', '--distribute', 'c:\\talos-slave\\test\\build\\venv']
00:25:37     INFO -  New python executable in c:\talos-slave\test\build\venv\Scripts\python.exe
00:25:37     INFO -  ERROR: The executable c:\talos-slave\test\build\venv\Scripts\python.exe is not functioning
00:25:37     INFO -  ERROR: It thinks sys.prefix is 'c:\\talos-slave\\test' (should be 'c:\\talos-slave\\test\\build\\venv')
00:25:37     INFO -  ERROR: virtualenv is not compatible with this system or executable
00:25:37     INFO -  Note: some Windows users have reported this error when they installed Python for "Only this user".  The problem may be resolvable if you install Python "For all users".  (See https://bugs.launchpad.net/virtualenv/+bug/352844)
00:25:37    ERROR - Return code: 100

See bug 700415 comment 46, 47, 68, 70 .

https://bugs.launchpad.net/virtualenv/+bug/352844/comments/1

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

would we want to just bite the bullet and install python 2.7 ?

Comment 3

[Aki Sasaki (not active)]

Yes, though we need to keep bug 602908 in sync.

Comment 4

[cmtalbert]

(In reply to Aki Sasaki [:aki] from comment #3)
> Yes, though we need to keep bug 602908 in sync.

Do you mean that 602908 requires 2.6, or that we just need to verify that the issue in 602908 is solved by either 2.7 or 2.6?

Comment 5

[William Lachance (:wlach)]

After discussing on irc with aki, ctalbert, jhammel, jmaher and armenzg, I think we're all agreed that 2.7 is the desired target here. That's what will really help us get where we want to go.

Comment 6

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to William Lachance (:wlach) from comment #5)
> After discussing on irc with aki, ctalbert, jhammel, jmaher and armenzg, I
> think we're all agreed that 2.7 is the desired target here. That's what will
> really help us get where we want to go.

Just wondering - Python 2.7.3 just got released, are we targeting this version of Python 2.7?

http://www.python.org/download/releases/2.7.3/

Comment 7

[William Lachance (:wlach)]

(In reply to Gary Kwong [:gkw, :nth10sd] from comment #6)
> (In reply to William Lachance (:wlach) from comment #5)
> > After discussing on irc with aki, ctalbert, jhammel, jmaher and armenzg, I
> > think we're all agreed that 2.7 is the desired target here. That's what will
> > really help us get where we want to go.
> 
> Just wondering - Python 2.7.3 just got released, are we targeting this
> version of Python 2.7?
> 
> http://www.python.org/download/releases/2.7.3/

I think the plan is to go with the latest/greatest version of python 2.7 when we do the upgrade, whatever that is.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

if 2.7.3 was just released, we should make sure all our tests work fine with that.

Comment 9

[Chris AtLee [:catlee]]

I'm not so concerned about the minor version here. If all our systems have/use 2.7.x by default I think that's a big win on its own.

Comment 10

[cmtalbert]

So, what is holding this back? So that we can get a clear set of steps forward. I think Will did a bunch of work to ensure that we can run on 2.7 w.r.t. removing the pywin32 dependency.

Comment 11

[William Lachance (:wlach)]

(In reply to Clint Talbert ( :ctalbert ) from comment #10)
> So, what is holding this back? So that we can get a clear set of steps
> forward. I think Will did a bunch of work to ensure that we can run on 2.7
> w.r.t. removing the pywin32 dependency.

IIRC there was some concern that upgrading the version of python 2.7 would affect Talos results, so we wanted to run some Talos tests in staging with python 2.7 (+ my pywin32 removal patch) to make sure the numbers aren't affected. Probably best to follow up on that in bug 734466.

I don't see why that should block *installing* python 2.7 on test machines though. We can always install it without using it.

Comment 12

[Armen [:armenzg]]

FTR rail deployed python2.7 to Windows XP and Windows 7 machines.
The path is C:\mozilla-build\python27\python.exe

Comment 13

[cmtalbert]

Wooot! Great, how do we ensure that Talos starts using it?

Comment 14

[Aki Sasaki (not active)]

Atm, mozharness talos will use py27 out of the box.
I think it would be difficult to allow buildbot (non-mh) talos to use it since it would have to switch to virtualenv creation to install pywin32.

Comment 15

[cmtalbert]

So then in other words, as we convert to mozharness we pick up python 2.7 and can drop that damn pywin32 requirement? :)

Comment 16

[Aki Sasaki (not active)]

We're installing pywin32 in mozharness because it's (aiui) required by Talos.
However, we won't be required to install pywin32 on the machine previous to running Talos in mozharness, as it will install it in the virtualenv for you.

But yes, py27 and no requirement of having pywin32 preinstalled on the test pool. I'm currently envisioning switching to mozharness Talos via train migration, so we may still support buildbot Talos for a few releases after m-c switches over.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

we have patches to remove our requirement of pywin32.  Once we have a train car running 100% mozharness talos we could land that patch on the specific branch!

Comment 18

[Aki Sasaki (not active)]

That may require a separate talos package with pywin32 removed from the setup.py dependencies, but yes, we could do that :)

Comment 19

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #14)
> Atm, mozharness talos will use py27 out of the box.

For clarity, atm *win32* mozharness talos will use py27 out of the box.
Leopard (EOL Fx17) will be using 2.5.1; every other platform will use either 2.6 or 2.7.

https://bugzilla.mozilla.org/show_bug.cgi?id=763550#c2

Comment 20

[William Lachance (:wlach)]

(In reply to Joel Maher (:jmaher) from comment #17)
> we have patches to remove our requirement of pywin32.  Once we have a train
> car running 100% mozharness talos we could land that patch on the specific
> branch!

Patch to remove pywin32 dependency is bug 726700.

Comment 21

[Jordan Lund (:jlund)]

FYI in case its unknown, I've stumbled on a dependency for win32api (pywin32)

currently our mouse-and-screen step here: http://hg.mozilla.org/build/buildbotcustom/file/78777283f1a2/process/factory.py#l4951

uses a script from here: http://hg.mozilla.org/build/tools/file/3b9ec354d891/scripts/support

This step is being ported to mozharness. We have switched to python 2.7 with windows, but this script fails as it can not import win32api

Comment 22

[William Lachance (:wlach)]

(In reply to Jordan Lund (:jlund) from comment #21)
> FYI in case its unknown, I've stumbled on a dependency for win32api (pywin32)
> 
> currently our mouse-and-screen step here:
> http://hg.mozilla.org/build/buildbotcustom/file/78777283f1a2/process/factory.
> py#l4951
> 
> uses a script from here:
> http://hg.mozilla.org/build/tools/file/3b9ec354d891/scripts/support
> 
> This step is being ported to mozharness. We have switched to python 2.7 with
> windows, but this script fails as it can not import win32api

This doesn't look like it would be terribly difficult to port to ctypes. I can take a look at it if no one else is willing/able.

Comment 23

[Geoff Lankow (:darktrojan)]

Any news here? I don't think I can go any further on bug 660788 without Popen.kill (new in Python 2.6).

Comment 24

[William Lachance (:wlach)]

(In reply to Geoff Lankow (:darktrojan) from comment #23)
> Any news here? I don't think I can go any further on bug 660788 without
> Popen.kill (new in Python 2.6).

I'll look into this after I finish my current pending task (hopefully later today).

If you want to jump ahead of me, you can use my pywin32->ctypes conversion patch in bug 726700 as a guide. Be sure to let us know if you do so we don't duplicate work. :)

Comment 25

[Armen [:armenzg]]

I was looking at what is needed for the Windows hosts and we need to deploy PyWin32 as python 2.7 is already deployed.

On another note, I spoke with jmaher and aki and we might be getting what we are aiming in here by just waiting on mozharness talos + unit tests which will be accomplished within this quarter (only 5 bugs away as per aki).

In fact, I'm very happy that mozharness is doing things right. Every script running on our machines should be capable of setting its dependencies correctly without off band deployments. This is bringing a whole new world of liberation to us all! rather than having to mess with PATHs and not shiny deployment methods (OPSI, ssh, RDP/VNC).

Fine if we close this and get things done properly with mozharness?
Right now we don't have resources to put into while mozharness is going forward with full steam.

[1] Windows 7
C:\Users\cltbld>C:\mozilla-build\python27\python.exe
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win32api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named win32api

[2]
C:\Documents and Settings\cltbld>C:\mozilla-build\python27\python.exe
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win32api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named win32api

Comment 26

[Joel Maher ( :jmaher ) (UTC -8)]

we would need to ensure the foopies are using 2.7.3 as well.  Are we using 2.7.3 explicitly for all tests?

Comment 27

[Justin Wood (:Callek)]

(In reply to Joel Maher (:jmaher) from comment #26)
> we would need to ensure the foopies are using 2.7.3 as well.

This part is tracked in Bug 902520

Comment 28

[Jonathan Griffin (:jgriffin)]

This has been completed for a while now.