804873 - Build with Python 2.7 on Windows build slaves

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Gregory Szorc [:gps]]

I pushed a try build (https://tbpl.mozilla.org/?tree=Try&rev=7179f7d5713f) requiring Python 2.7 to build the tree and it appears that the Windows builders are all building with Python 2.6.

Please update the Windows builders to build the tree with Python 2.7.

I'm not sure what goes into updating the Windows builders. But, if you put a python2.7.exe or 2.7 version of python.exe on $PATH before whatever is there now, that should be all that is needed for configure to pick up the newer Python.

Comment 1

[Armen [:armenzg]]

Adding coop as he might know what is needed.

Comment 2

[Chris Cooper [:coop] (he/him)]

IIRC correctly, the version of python that gets used (2.6) lives in D:\mozilla-build\python. We should be able to simply change the PATH.

Let me try some things in staging.

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #2)
> IIRC correctly

Sigh. I need coffee badly.

Comment 4

[Chris Cooper [:coop] (he/him)]

Python 2.7 is, in fact, installed on these slaves already, but the pywin32 extensions for it are not. This was causing builds to fail out early in the setup.

I've figured out a method to install the win32 extensions silently as Administrator, and am now running a build through staging to make sure the build itself is unaffected. 

Aside from the extensions install, I've only needed to move python27 to the front of our PATHs on Windows in the buildbot-configs so far.

Comment 5

[(not currently active) Ted Mielczarek]

Nothing in the build itself requires pywin32. Is this something in the automation?

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> Nothing in the build itself requires pywin32. Is this something in the
> automation?

Yes, I believe the pre-build file removal/cleanup steps on Windows rely on those apis.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #4) 
> I've figured out a method to install the win32 extensions silently as
> Administrator, and am now running a build through staging to make sure the
> build itself is unaffected. 

I've deployed this to all the w64 builders. I'm now checking whether we need this for the w32 builders.

Should have the configs patch ready this afternoon.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #7) 
> I've deployed this to all the w64 builders. I'm now checking whether we need
> this for the w32 builders.

Deployed on the w32 slaves now too.

Working on the configs patch now.

Comment 9

[Chris Cooper [:coop] (he/him)]

I've added the install notes to the refplatform docs:

https://wiki.mozilla.org/ReferencePlatforms/Win64#Python_2.7.3
https://wiki.mozilla.org/ReferencePlatforms/Win32#Install_pywin32

Comment 10

[Chris Cooper [:coop] (he/him)]

Attachment: Add python 2.7 ahead of the buildbotve version of python on win32 and win64

I've done successful dep/nightly builds on win64 and l10n repacks on win32 using this patch.

Comment 11

[Armen [:armenzg]]

Comment on attachment 683741 [details] [diff] [review]
Add python 2.7 ahead of the buildbotve version of python on win32 and win64

Review of attachment 683741 [details] [diff] [review]:
-----------------------------------------------------------------

Excellent!

Comment 12

[Chris Cooper [:coop] (he/him)]

Comment on attachment 683741 [details] [diff] [review]
Add python 2.7 ahead of the buildbotve version of python on win32 and win64

https://hg.mozilla.org/build/buildbot-configs/rev/fc70444b37f7

Comment 13

[Chris Cooper [:coop] (he/him)]

Second times the charm. Here's a log from one of Ehsan's check-ins today that shows python 2.7 being used successfully to purge_builds at the start of a run:

https://tbpl.mozilla.org/php/getParsedLog.php?id=17309992&tree=Mozilla-Inbound&full=1

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #13)
> Second times the charm. Here's a log from one of Ehsan's check-ins today
> that shows python 2.7 being used successfully to purge_builds at the start
> of a run:

And for the rest of the build process as well, natch. Is my releng bias showing?

Comment 15

[Gregory Szorc [:gps]]

Try is still on 2.6:

https://tbpl.mozilla.org/?tree=Try&rev=c5d14c3a9e17

Comment 16

[Chris Cooper [:coop] (he/him)]

Here's a rebuild of Greg's failed try push that's working now:

https://tbpl.mozilla.org/?tree=Try&rev=c5d14c3a9e17&jobname=WINNT%205.2%20try%20build

Comment 17

[Amy Rich [:arr] [:arich]]

There was never a bug filed to have this rolled into the automated image, and the attempt to do that now is failing.  Bug 827497 is tracking that work.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

From what I can gather the story is this:

 * Python-2.7.3 + pywin32 was installed by hand on all w64 systems in this bug

 * That was not reflected in the MDT task sequence

 * As a result, systems built with the new MDT task sequence don't work, and we're iterating through trying to get them to work in bug 827497.  Hopefully learning something about how Python is used in the Build process while doing so!

So the only remaining action here is to close 827497.

Comment 19

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] from comment #18)
> From what I can gather the story is this:
> 
>  * Python-2.7.3 + pywin32 was installed by hand on all w64 systems in this
> bug
> 
>  * That was not reflected in the MDT task sequence

It was reflected in our reference platform docs, that aiui was the baseline for MDT task sequence for this

https://wiki.mozilla.org/ReferencePlatforms/Win64#Python_2.7.3

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Turns out we figured out enough of the details to get this installed.