1315977 - Mac build machines doing l10n repacks need python 2.7.6 or newer

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Axel Hecht [:Pike]]

We have a new version of compare-locales that I'd like to use in mozilla-central.

That has python 2.7.6 as minimum version requirement, do we have that on all automation that does l10n repacks?

Bug 1315976 tracks the uplift into mozilla-central.

Comment 1

[Axel Hecht [:Pike]]

The mac slaves are burning now that the update to compare-locales landed.

Can we upgrade the python version on those quickly?

Comment 2

[Justin Wood (:Callek)]

Specifically this is a regex bug with py < 2.6

>>> re.compile('(?P<post>\s*)?', re.DOTALL)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tools/python27/lib/python2.7/re.py", line 190, in compile
    return _compile(pattern, flags)
  File "/tools/python27/lib/python2.7/re.py", line 242, in _compile
    raise error, v # invalid expression
sre_constants.error: nothing to repeat
>>> 
[[e-mail] ~]$ /tools/python27/bin/python2.7 --version
Python 2.7.3

The same works locally for me, on linux, with py2.10

example log: https://treeherder.mozilla.org/logviewer.html#?job_id=5714968&repo=mozilla-central#L2480

Comment 3

[Justin Wood (:Callek)]

(In reply to Axel Hecht [:Pike] from comment #1)
> The mac slaves are burning now that the update to compare-locales landed.
> 
> Can we upgrade the python version on those quickly?

Ok, looking at the (lack of) clean docs for upgrading python on Lion, it appears we can't easily upgrade it this week.

Before Hawaii is a tough time to do potentially tree-closing fragile upgrades, especially with little notice.

Is this something we can (easily?) hotpatch in compare-locales?

Comment 4

[Justin Wood (:Callek)]

(Self Notes, https://bugzilla.mozilla.org/show_bug.cgi?id=602908#c20 -- https://bugzilla.mozilla.org/show_bug.cgi?id=882869 )

Comment 5

[Axel Hecht [:Pike]]

Nah, there's no easy way to fix this in compare-locales code. That code has complex regexes and small changes produce weird results.

I'll need to back out for now until this is fixed. IOW, I'd really prefer for our build automation to use a recent version of python 2.7, compared to having to maintain an environment for python development that's broken and possibly unsecure.

Comment 6

[Justin Wood (:Callek)]

Additional self-note: https://bugzilla.mozilla.org/show_bug.cgi?id=985923 (which seems to have enabled py2.7.7 on linux inside Mock.)

:coop, how do we feel about trying to get buildduty to investigate upgrading to at least 2.7.7 on OSX Builders, and a timeline for that?

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Justin Wood (:Callek) from comment #6) 
> :coop, how do we feel about trying to get buildduty to investigate upgrading
> to at least 2.7.7 on OSX Builders, and a timeline for that?

In the absence of cross-compiled l10n repacks (which also needs to be addressed), this will need to be fixed. I'll add it to the buildduty queue.

Comment 9

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

I have loaned a bld-lion-r5 machine which I pinned to my puppet enviornment, created python27-2.7.12-1.dmg which I manually copy to puppet master,
In my enviornment I setup puppet to use python27-2.7.12-1.dmg and upgraded python to version 2.7.12 on the machine that I loaned.

[[e-mail] bin]$ ./python --version
Python 2.7.12

I did some tests,I ran the scripts from each step on the l10n-osx job that failed on this machine and on my side those errors were not present anymore.
Maybe there are also some tests needed before we can use this dmg file in production as I was only able to test it whithout connecting the loaner machine to a master, so what should be the next step here?

Comment 10

[Justin Wood (:Callek)]

(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #9)
> I have loaned a bld-lion-r5 machine which I pinned to my puppet enviornment,
> created python27-2.7.12-1.dmg which I manually copy to puppet master,
> In my enviornment I setup puppet to use python27-2.7.12-1.dmg and upgraded
> python to version 2.7.12 on the machine that I loaned.
> 
> [[e-mail] bin]$ ./python
> --version
> Python 2.7.12
> 
> I did some tests,I ran the scripts from each step on the l10n-osx job that
> failed on this machine and on my side those errors were not present anymore.
> Maybe there are also some tests needed before we can use this dmg file in
> production as I was only able to test it whithout connecting the loaner
> machine to a master, so what should be the next step here?

Lets make sure that buildbot starts up on a fresh boot just fine, and then we can check for a few days (maybe) if it works for most jobs on production trees.

If it does, then we can disable again, re-image entirely and land the puppet patch to production. If it doesn't sheriffs can disable and we can return to the drawing board.

If this system is using py2.7.12 I think we'll be fine here, the sad part is the manual l10n steps may not have worked as a good validation, due to the potential that running them updated the tree away from the failed rev to a newer revision.

Thanks!

Comment 11

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppet.patch

Attached is the patch for Puppet to use python27-2.7.12-1-test.dmg for Darwin systems, the location for this file is in /data/repos/DMGs/10.7
on releng-puppet2.srv.releng.scl3.mozilla.com hostname.

Comment 12

[Justin Wood (:Callek)]

Comment on attachment 8819824 [details] [diff] [review]
bug1315977_puppet.patch

Review of attachment 8819824 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/packages/manifests/mozilla/python27.pp
@@ -89,4 @@
>                      Anchor['packages::mozilla::python27::begin'] ->
>                      packages::pkgdmg {
>                          python27:
> -                            version => "2.7.3-1";

This won't work here, since you only built/tested python 2.7.12 on builders, you'll need logic similar to:

https://dxr.mozilla.org/build-central/source/puppet/modules/packages/manifests/mozilla/screenresolution.pp#16

You'll also want the os_version_specific => true, flag for the 2.7.12 version, so it looks in the 10.7 directory. (e.g. like: https://dxr.mozilla.org/build-central/source/puppet/modules/packages/manifests/puppet.pp#48 )

Also please do the testing as I suggested in the previous comment :-)

Comment 13

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: Bug1315977_puppet.patch

Attached is the patch for Puppet to use python27-2.7.12-1-test1.dmg for Darwin production version 10.7 only.

Comment 14

[Justin Wood (:Callek)]

Comment on attachment 8823588 [details] [diff] [review]
Bug1315977_puppet.patch

Review of attachment 8823588 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/packages/manifests/mozilla/python27.pp
@@ +97,5 @@
> +                        '10.7': {
> +                            Anchor['packages::mozilla::python27::begin'] ->
> +                            packages::pkgdmg {
> +                                python27:
> +                                    version => "2.7.12-1-test1";

seeing -test1 makes me *think* it shouldn't be in production, but I suspect this was named this way to test the patch/python itself.

Can we rename it to not have that connotation before we land?

You're also missing the os_version_specific flag as mentioned in a comment above, that I think we need here. (c.f. c#11 for why I think its likely os version specific, being in the 10.7 folder, that flag will make puppet look in that folder for the binary) - this is the only reason for r-

Comment 15

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: Bug1315977_puppet.patch

recreated patch

Comment 16

[Kim Moir [:kmoir] ET]

Andrei is having trouble testing this patch.  I've been looking at it too and am stuck.  The issue is that with this patch is that the worker machine connects to the master, and doesn't run buildbot. If you start buildbot manually, a job runs, the worker reboots and then buildbot doesn't stop. Looked at the logs, not sure what this is occurring.  An example of this is bld-lion-r5-070.build.releng.scl3.mozilla.com which is pinned to Andrei's puppet env.  Anyone else have any debugging suggestions?

Comment 17

[Kim Moir [:kmoir] ET]

from irc
Callek> kmoir: so, re: python on osx -- 070 looks funky -- https://papertrailapp.com/systems/bld-lion-r5-070/events?focus=755883677212381192 specifically around tasks.dreset does not exist, and mig crashing, and seeming other error codes...
1:23 PM 
<kmoir> Kim Moir yeah, I saw the mig crashing
1:25 PM perhaps he should compare the logs on a non-updated machine and the updated machine 
1:25 PM 
<Callek> kmoir: so the twistd.log has a BUNCH of : 
1:26 PM https://irccloud.mozilla.com/pastebin/O4HwAaD3/
 Plain Text • 12 lines raw | line numbers 
1:26 PM that is on the 070 slave
1:26 PM 
— Callek isn't sure why it can't connect, but it *could* be unable to due to python networking/ssl support changes in the newer python
1:27 PM 
<kmoir> Kim Moir hmm, that could be possible
1:27 PM 
<Callek> I'm completely not certain though
1:27 PM would be worth checking the twistd.log after you spin up a new buildbot though (or one already spins up) to see if its indeed connecting and running a job

Comment 18

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Did some tests,put in my enviornment version 2.7.3, re-imaged a machine and pinned it back to my enviornment but buildbot still don't start when machine is rebooted.Also manually started buildbot but at some point machine get rebooted,job interrupted and buildbot doesn't start automatically when the machine is up again(see last interrupted jobs for bld-lion-r5-085).

As we have the same situation (buildbot not starting automatically) when machine is configured with python 2.7.3 and with 2.7.12, I don't think the machine is unable to connect due to python networking/ssl support changes in the newer python version.

Comment 19

[Kim Moir [:kmoir] ET]

So to clarify, are you saying that a newly imaged bld-lion-r5 machine won't consistently stay up with buildbot running when locked to a master? Even when not pinned to your puppet env?

Comment 20

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

The machine will stay up with buildbot running when the machine is not pined to my enviornment, the problem appear when the machine is pinned to my puppet enviornment,after that buildbot will not ran automatically anymore.

Comment 21

[Kim Moir [:kmoir] ET]

Okay, looking at the log for 

bld-lion-r5-070.build.releng.scl3.mozilla.com
[[e-mail] slave]# grep -ir "SlaveBuilder._ackFailed" twistd.log
2017-01-11 09:04:34-0800 [Broker,client] SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate
2017-01-11 09:04:34-0800 [Broker,client] SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate
2017-01-11 09:04:34-0800 [Broker,client] SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate
2017-01-11 09:04:34-0800 [Broker,client] SlaveBuilder._ackFailed: sendComplete
2017-01-11 09:04:34-0800 [Broker,client] SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate

this seems to indicate that the worker machine is losing connection with it's master.  

Looking at [e-mail] which is not pinned to your puppet env, it looks this message doesn't appear it it's logs. I wonder if if there is a problem with the new version of python and the very old version of buildbot we have on the machine.

Comment 22

[Phil Ringnalda (:philor)]

Disabled 060 and 070, since they were taking production jobs and burning everything with https://treeherder.mozilla.org/logviewer.html#?job_id=69327911&repo=autoland

Comment 23

[Phil Ringnalda (:philor)]

And 065.

Comment 24

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: error.png

Did some other tests while Buildbot refuse to start when the machine is rebooted while is pinned to my puppet enviornment.
So after I pinned the machines to my enviornment and upgraded Python to version 2.7.12, I pinned the machines to production enviornment because Python will remain with version 2.7.12 (can't downgrade) and with these options Buildbot will start automatically.
These machines(bld-lion-r5-060,070,065 and 085) used Python 2.7.12 while ran some jobs so I could monitored the activity, and the result is that most of them failed with this error (1),mostly on this step (2),it failed to create virtualenv (3).

Callek do you have any recommendation for this situation?

(1) error.png
(2)step.png  /tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py 
(3) https://hg.mozilla.org/mozilla-central/file/tip/python/mozbuild/mozbuild/virtualenv.py#l203

Comment 25

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: step.png

Comment 26

[Justin Wood (:Callek)]

Greg, any chance you can lend some insight here?

Running on a patched ver of the source tree, (adding XXXCallek lines above the venv check at https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/virtualenv.py#125 ) I got confirmation that attachment 8827387 [details] is showing an error here...   [Note, I dropped __file__ from the venv check on line 117 to facilitate the debug printing]


[[e-mail] ~]$ /tools/buildbot/bin/python /builds/slave/autoland-m64-00000000000000000/build/src/mach python /builds/slave/autoland-m64-00000000000000000/build/src/config/printconfigsetting.py /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/dist/bin/application.ini App SourceStamp
XXXCALLEK ==== Check python binary
XXXCallek ==== Python /tools/buildbot/bin/python
XXXCallek ==== Python_path /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/_virtualenv/bin/python
Error running mach:

    ['python', '/builds/slave/autoland-m64-00000000000000000/build/src/config/printconfigsetting.py', '/builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/dist/bin/application.ini', 'App', 'SourceStamp']

.....

==============

Once I did a rm -rf of the _virtualenv and re-ran I got the following issue, which I'm not sure where/why its being hit.


[[e-mail] ~]$ /tools/buildbot/bin/python /builds/slave/autoland-m64-00000000000000000/build/src/mach python /builds/slave/autoland-m64-00000000000000000/build/src/config/printconfigsetting.py /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/dist/bin/application.ini App SourceStamp
Using real prefix '/tools/python27'
New python executable in /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/_virtualenv/bin/python
Overwriting /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/_virtualenv/lib/python2.7/distutils/__init__.py with new content
Traceback (most recent call last):
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/virtualenv/virtualenv.py", line 2325, in <module>
    main()
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/virtualenv/virtualenv.py", line 711, in main
    symlink=options.symlink and hasattr(os, 'symlink')) # MOZ: Make sure we don't use symlink when we don't have it
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/virtualenv/virtualenv.py", line 926, in create_environment
    install_distutils(home_dir)
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/virtualenv/virtualenv.py", line 1487, in install_distutils
    writefile(os.path.join(distutils_path, '__init__.py'), DISTUTILS_INIT)
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/virtualenv/virtualenv.py", line 371, in writefile
    with open(dest, 'wb') as f:
IOError: [Errno 13] Permission denied: '/builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/_virtualenv/lib/python2.7/distutils/__init__.py'
Error running mach:

    ['python', '/builds/slave/autoland-m64-00000000000000000/build/src/config/printconfigsetting.py', '/builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/dist/bin/application.ini', 'App', 'SourceStamp']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Failed to create virtualenv: /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/_virtualenv

  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/mach_commands.py", line 41, in python
    self._activate_virtualenv()
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/mozbuild/mozbuild/base.py", line 623, in _activate_virtualenv
    self.virtualenv_manager.ensure()
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/mozbuild/mozbuild/virtualenv.py", line 170, in ensure
    return self.build(python)
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/mozbuild/mozbuild/virtualenv.py", line 443, in build
    self.create(python)
  File "/builds/slave/autoland-m64-00000000000000000/build/src/python/mozbuild/mozbuild/virtualenv.py", line 207, in create
    'Failed to create virtualenv: %s' % self.virtualenv_root)

Comment 27

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #26)
> Greg, any chance you can lend some insight here?

... noteworthy, if I run the step via the system python directly, rather than buildbot's, everything works!

[[e-mail] ~]$ /tools/python27/bin/python2.7 /builds/slave/autoland-m64-00000000000000000/build/src/mach python /builds/slave/autoland-m64-00000000000000000/build/src/config/printconfigsetting.py /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/dist/bin/application.ini App SourceStamp
New python executable in /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/_virtualenv/bin/python2.7
Also creating executable in /builds/slave/autoland-m64-00000000000000000/build/src/obj-firefox/_virtualenv/bin/python
Installing setuptools, pip, wheel...done.
running build_ext
copying build/lib.macosx-10.4-x86_64-2.7/psutil/_psutil_osx.so -> psutil
copying build/lib.macosx-10.4-x86_64-2.7/psutil/_psutil_posix.so -> psutil

70db86990f0df9766d1eb582bd7668a1dd6983bb



The buildbot venv is 2.7.12 and seems to have been [re]created yesterday (after the py2.7.12 was installed)

Comment 28

[Gregory Szorc [:gps]]

That "permission denied" failure regarding distutils rang a bell. So I dug up bug 758694. Can you read through that bug and needinfo me again if the theory of multiple python installs interfering with each other doesn't pan out?

Comment 29

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Found some symbolic links by checking the logs that are different on the machine where I upgraded python and the others,these can be seen under: 
"/builds/slave/m-cen-m64-d-000000000000000000/build/src/obj-firefox/_virtualenv/bin"

[[e-mail] bin]# (python upgraded to version 2.7.12)
lrwxrwxr-x  1 cltbld  staff     6 Jan 18 07:58 python2 -> python
lrwxrwxr-x  1 cltbld  staff     6 Jan 18 07:58 python2.7 -> python
lrwxrwxr-x  1 cltbld  staff     9 Jan 18 06:50 python -> python2.7

[[e-mail] bin]# (python version 2.7.3)
lrwxrwxr-x  1 cltbld  staff     9 Jan 16 12:11 python -> python2.7
lrwxrwxr-x  1 cltbld  staff     9 Jan 16 12:11 python2 -> python2.7
-rwxrwxr-x  1 cltbld  staff  9012 Jan 16 12:11 python2.7

So basically on the machine which was pinned to my enviornment, python2.7 is pointed to python (python2.7 -> python) and python -> python2.7 .Not sure if this is related to bug 758694 but could explain the message error: "OSError: [Errno 62] Too many levels of symbolic links".

In this case what should be our move in fixing this issue?

Comment 30

[Kim Moir [:kmoir] ET]

Callek looked at this today, and doesn't know a solution, he suggested mshal might be a good person to talk to.

Comment 31

[Jonathan Griffin (:jgriffin)]

mshal, can you take a look at this? gps is going to be unavailable for a while

Comment 32

[Michael Shal [:mshal]]

Sure, I can try to take a look later today.

Comment 33

[Michael Shal [:mshal]]

When you use the new version, is python invoked as 'python2.7' or something instead of 'python'?

I can reproduce the symlink loop issue in the _virtualenv/bin directory as follows (just calling virtualenv.py directly for simplicity):

$ python python/virtualenv/virtualenv.py --no-download obj-test/_virtualenv

$ ls -l obj-test/_virtualenv/bin/python*
-rwxrwxr-x 1 mshal mshal 3777000 Feb 10 16:11 obj-test/_virtualenv/bin/python
lrwxrwxrwx 1 mshal mshal       6 Feb 10 16:11 obj-test/_virtualenv/bin/python2 -> python
lrwxrwxrwx 1 mshal mshal       6 Feb 10 16:11 obj-test/_virtualenv/bin/python2.7 -> python
-rwxrwxr-x 1 mshal mshal    2370 Feb 10 16:11 obj-test/_virtualenv/bin/python-config

$ python2.7 python/virtualenv/virtualenv.py --no-download obj-test/_virtualenv

$ ls -l obj-test/_virtualenv/bin/python*
lrwxrwxrwx 1 mshal mshal    9 Feb 10 16:11 obj-test/_virtualenv/bin/python -> python2.7
lrwxrwxrwx 1 mshal mshal    6 Feb 10 16:11 obj-test/_virtualenv/bin/python2 -> python
lrwxrwxrwx 1 mshal mshal    6 Feb 10 16:11 obj-test/_virtualenv/bin/python2.7 -> python

Note that the first time we invoke with 'python' and the second time with 'python2.7'. This impacts sys.executable, which is how virtualenv.py determines which files to create and which files to symlink in the bin directory.

virtualenv.py basically does this:

1) Copy sys.executable to bin/[sys.executable]
2) Create symlinks for 'python', 'python2', and 'python2.7', except for whichever of these matches sys.executable

So in the first invocation, it creates 'python' as a regular file, then 'python2' and 'python2.7' as symlinks. If we keep the virtualenv around and then invoke it again with 'python2.7', it skips copying sys.executable to python2.7 because it already exists, and then overwrites 'python' and 'python2' with symlinks pointing to python2.7

To me this seems like a bug in virtualenv.py, though I think we can work around it by running python as "python" in all cases. I'm not sure how installing the package would have changed that though. Any ideas there?

Comment 34

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

(In reply to Michael Shal [:mshal] from comment #33)
> When you use the new version, is python invoked as 'python2.7' or something
> instead of 'python'?

I think python is invoked as 'python'(this can be seen on attached file error.png) 


> To me this seems like a bug in virtualenv.py, though I think we can work
> around it by running python as "python" in all cases. I'm not sure how
> installing the package would have changed that though. Any ideas there?

The problem is that in the first place we have this symlink structure:
  - python -> python2.7
  - python2 -> python2.7
  - python2.7
And after updating python to version 2.7.12 we have the one bellow and the step to create the virtual enviornment failed.
  - python2 -> python
  - python2.7 -> python
  - python -> python2.7

If we could ran puppet directly with python version 2.7.12 for OSX there will be no conflict as no other virtual enviornment previous created would exist but we could encounter other issues. Callek also tried to remove the virtual enviornment created with python version 2.7.3 in comment 26 and to re-run with buildbot but still could not create the enviornment.

If python was already ran as "python" in all cases then we must find another solution to work around it,If you have any suggestions please let me know.

Comment 35

[Michael Shal [:mshal]]

Can you point me to the python packages that we use? Both the old (2.7.3?) and new (2.7.12). ie: the actual .dmg files or whatever we use to install them on the machines.

Comment 36

[Michael Shal [:mshal]]

:ewong, it looks like you hit this same virtualenv symlink loop in https://github.com/pypa/virtualenv/pull/892 - what did you end up doing instead?

Comment 37

[Michael Shal [:mshal]]

And FWIW, this bug still exists in 15.1.0 of virtualenv (we use 15.0.1 currently).

Comment 38

[Edmund Wong (:ewong)]

(In reply to Michael Shal [:mshal] from comment #36)
> :ewong, it looks like you hit this same virtualenv symlink loop in
> https://github.com/pypa/virtualenv/pull/892 - what did you end up doing
> instead?

hi Michael,

I need to dig up the bug that I worked on as I don't remember which bug
this was from; however, I do remember attempting to do a upstream
PR along the lines of:

https://github.com/pypa/virtualenv/compare/master...ewongbb:chksymlinks

However that didn't go anywhere.

Comment 39

[Edmund Wong (:ewong)]

I think I found the relevant bug:

Bug 1262760

Comment 40

[Edmund Wong (:ewong)]

Basically the solution is to explicitly specify the full python path to
the _virtualenv.

Comment 41

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

We use DMG files which are located in releng-puppet2.srv.releng.scl3.mozilla.com hostname.
The path for the DMG files that are used here (1) are located in:

-/data/repos/DMGs/10.7/python27-2.7.12-1.dmg for (2.7.12)
-/data/repos/DMGs/10.7/python27-2.7.3-1.dmg  for (2.7.3)

If you want to do some test please feel free to use any of the these machines :
-bld-lion-r5-060
-bld-lion-r5-070
-bld-lion-r5-085
-bld-lion-r5-065

(1) https://dxr.mozilla.org/build-central/source/puppet/modules/packages/manifests/mozilla/python27.pp#92

Comment 42

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Bellow is the link to the stdio output of step.png:
https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64/1484583681/autoland-macosx64-bm82-build1-build1626.txt.gz

Comment 43

[Michael Shal [:mshal]]

I was not able to reproduce this locally using the .dmg files - both seem to have the same file layout. I was thinking maybe one of them had a python2.7 binary and the other had a python binary or something, but that's not the case.

However I was able to continue Callek's line of thought in #c26 and #c27. The problem results from a few things:

1) We have two different pythons that are used 
  - mozharness uses /tools/buildbot/bin/python 
  - configure uses the python2.7 it finds in the path [1], which is /tools/python/bin/python2.7

2) We have two different entrypoints into the same objdir/_virtualenv environment
  - one is from configure, which is called during 'mach build' [2]
  - the other is from mach directly when using 'mach python' [3]

Even though mozharness calls 'mach build' and later 'mach python' using the same /tools/buildbot/bin/python path, during 'mach build' we shell out to make and into configure, which determines the python to use by finding python2.7 in the PATH. So configure actually generates the virtualenv the first time using /tools/python/bin/python2.7, which sets up python2.7 as a regular file and 'python' and 'python2' as symlinks pointing to python2.7, as described in #c33.

After the build, mozharness now runs 'mach python' with the buildbot python, and tries to activate the virualenv. As part of this process, it does a check to see if the virtualenv needs to be rebuilt [4]. In this case, although the hex version matches (34016496), the size of the binaries is different (9012 vs 9116):

bld-lion-r5-070$ ls -l /tools/buildbot/bin/python /tools/python/bin/python2.7
-rwxr-xr-x  1 root  admin  9012 Jan 16 07:26 /tools/buildbot/bin/python
-rwxr-xr-x  1 root  wheel  9116 Dec 15 05:31 /tools/python/bin/python2.7

Since the sizes are different, it tries to rebuild the virtualenv using the buildbot python, only now sys.executable is just 'python' instead of 'python2.7' like we had when it was invoked from configure. This triggers the bug in the upstream virtualenv.py code, and we end up with a cycle of symlinks pointing at each other.

The reason we don't hit this on other builds is that the /tools/buildbot version of python is the same size as the /tools/python version:

bld-lion-r5-005$ ls -l /tools/buildbot/bin/python /tools/python/bin/python2.7
-rwxr-xr-x  1 root  admin  9012 Mar 30  2016 /tools/buildbot/bin/python
-rwxrwxr-x  1 root  wheel  9012 Jul 11  2013 /tools/python/bin/python2.7

So even though we're using a different python during 'mach python', it doesn't think the virtualenv is out of date and so it doesn't try to rebuild it.

As for a solution, we should probably get rid of one of the python installations, and stick with just /tools/buildbot/bin or /tools/python/bin in the PATH. Then both buildbot and configure and anything else that looks for a python will find the same one.

If that proves too difficult, it might be easier to install the same python from the 2.7.12 dmg into /tools/buildbot (I'm not sure how that gets installed in the first place - where does it come from?). Since we are moving away from buildbot anyway, it might be better to just go with a short-term solution that makes the sizes match.

In case it helps, I ran 'strings' on both versions of python that we use, and these are the differences:

$ diff buildbot-strings.txt python-strings.txt
33c33
< /var/root/build/Python-2.7.3/./Modules/
---
> /Users/cltbld/dmg-script/build/Python-2.7.12/./Modules/
35c35
< /private/var/root/build/Python-2.7.3/Modules/python.o
---
> /Users/cltbld/dmg-script/build/Python-2.7.12/Modules/python.o
36a37
> /Users/cltbld/dmg-script/build/Python-2.7.12/./Modules/python.c

I'm not sure why the buildbot one has strings for 'Python-2.7.3' in it, when /tools/buildbot/bin/python --version shows 'Python 2.7.12'

In any case, if you can get those two binaries to match it should fix the issue.

[1] https://dxr.mozilla.org/mozilla-central/source/configure#25
[2] https://dxr.mozilla.org/mozilla-central/rev/0eef1d5a39366059677c6d7944cfe8a97265a011/build/moz.configure/init.configure#192
[3] https://dxr.mozilla.org/mozilla-central/rev/0eef1d5a39366059677c6d7944cfe8a97265a011/python/mozbuild/mozbuild/virtualenv.py#166
[4] https://dxr.mozilla.org/mozilla-central/rev/0eef1d5a39366059677c6d7944cfe8a97265a011/python/mozbuild/mozbuild/virtualenv.py#137

Comment 44

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Based on Michael's investigation in #c43 it seems that the /tools/buildbot/bin/python still has 'strings' from previous python version 2.7.3 even after upgrade and even if the version is shown as 2.7.12.
I did some tests and found a possible workaround,right after re-image I delete /tools/buildbot and I upgrade python, after this /tools/buildbot/bin/python will no longer contain strings from the previous version and by enabling bld-lion-r5-065 it seems the jobs will not fail any more.
However this solution require a downtime for the machines in this pool (bld-lion-r5) and eventually we will need in future to do the same to the test machines and things can get more complicated there.
Anyone with a better idea is more than welcome to share it with us.

Comment 45

[Chris AtLee [:catlee]]

I think we need a way for the buildslave puppet module to recreate the virtualenv when the python package changes.

https://hg.mozilla.org/build/puppet/file/tip/modules/buildslave/manifests/install.pp#l30

Comment 46

[Axel Hecht [:Pike]]

I think I found a way to get compare-locales going without the python upgrade, but I still think it'd be good to get a current version of python 2.7 running across our build platforms, so I'd not close this one.

Comment 47

[Kim Moir [:kmoir] ET]

Axel, is this bug still needed?  Coop mentioned you found a workaround.

Comment 48

[Axel Hecht [:Pike]]

I don't need this for compare-locales updates anymore, no. Just got a newer version on mozilla-central and nightlies that works on 2.7.x <= 5, too.

Comment 49

[Kim Moir [:kmoir] ET]

Andrei: catlee asked me about this bug earlier this week.  Did you investigate a way to delete the old files via puppet?

Comment 50

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppetv2.patch

Puppet patch to delete the old /tools/buildbot before applying the new version.
Tested on my environment and it seems the strings from the older version(2.7.3) are gone.

Comment 51

[Justin Wood (:Callek)]

Comment on attachment 8864144 [details] [diff] [review]
bug1315977_puppetv2.patch

Review of attachment 8864144 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/packages/manifests/mozilla/python27.pp
@@ +110,5 @@
> +                        '10.7': {
> +                            exec {"remove tools/buildbot":
> +                                path => "/tools/",
> +                                command => "/bin/rm -rf /tools/buildbot",
> +                                onlyif => '/bin/test -d /tools/buildbot',

this will remove the dir every single run, not really what we want.

It also has no ordering dep.. which means that this could happen in following order (in theory):

* Create buildbot dir
* Delete Buildbot Dir
* Install py 2.7.12

- or -

* Create Buildbot Dir
* Install Py 2.7.12
* Delete Buildbot Dir

- or -

* Delete [not really, doesn't exist] buildbot dir
* Create buildbot dir
* install py 2.7.12

Comment 52

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: Bug1315977_puppetv3.patch

Created patch for puppet which will delete old /tools/buildbot,then will install the new python version and after that it will create a file /tools/updated.txt.
As long as this file exist,on the next puppet run, /tools/buildbot will not be deleted again.
Also the ordering is ok,we will not encounter situation like in #c51.

Kim I asked a review from you since Callek is on PTO but if you want to change to someone else there is no problem.

Comment 53

[Kim Moir [:kmoir] ET]

Comment on attachment 8868991 [details] [diff] [review]
Bug1315977_puppetv3.patch

Looks like the flag strategy was able to work out

Please include the bug number in your patch for others who read the code to reference

Is there a way to use the file resource type to delete the files instead of rm -rf?

How did you test this patch?

Did you assign yourself a loaner build machine, apply the test to your machine and then reboot it a few times and look at the contents and the logs.  Could buildbot start and connect to a dev-master after the puppet change?

Comment 54

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Yes I tested by assigning some loaners build machines ,pined them to my enviornment ,applied the tests by running puppet with  -- debug and test to see if by running puppet, /tools/buildbot is deleted and the file updated.txt is created,and if the file exist then /tools/buildbot is not deleted.

Some older tests that I've done in the past showed that buildbot can start and can connect to a dev-master after the puppet change with this package but to be 100% sure we can test maybe on jamun branch to not interfere with the production.

I will also  add the bug number and search for a solution to use the file resource type to delete the files instead of rm -rf.

Comment 55

[Kim Moir [:kmoir] ET]

The jamun branch is being used for devedition work so not sure if this is the best place to test a puppet patch.  If it works with your testing and you can connect to a dev-master and run a build that is probably sufficient.

Comment 57

[Justin Wood (:Callek)]

Comment on attachment 8868991 [details] [diff] [review]
Bug1315977_puppetv3.patch

Review of attachment 8868991 [details] [diff] [review]:
-----------------------------------------------------------------

I'm r-'ing for the comments I put here (puppet ordering concerns are not as straight forward as I'd like).

I'm also sending off a feedback to Jake incase he has thoughts/ideas on a good way to do this need.

::: modules/packages/manifests/mozilla/python27.pp
@@ +111,5 @@
> +                            exec {"remove tools/buildbot":
> +                                path => "/tools/",
> +                                command => "/bin/rm -rf /tools/buildbot",
> +                                onlyif => '/bin/test -d /tools/buildbot && /bin/test ! -f /tools/updated.txt',
> +                                creates => '/tools/updated.txt' 

note: This `exec` doesn't create the file, that is created elsewhere (below)

@@ +120,5 @@
> +                                    os_version_specific => true,
> +                                    version => "2.7.12-1",
> +                                    before => Exec["remove tools/buildbot"];
> +                            }  -> Anchor['packages::mozilla::python27::end'] ->
> +                            file { "/tools/updated.txt":

I'd name this something other than "updated.txt" maybe "python_update.txt"

You can also use puppet NOTIFY/SUBSCRIBE support to do the rm and buildbot creation stuff in the right order.

The concern I have on that ordering is about the potential for us to have 10.7 systems that don't do buildbot. Or if the puppet inherent order changes for some unrelated reason.

---

Note that even though this `file` directive comes later in the manifest here, it is not guaranteed to run after the `exec` above, meaning it could run before the exec and it additionally is not guaranteed to run after the python upgrade. so it could get ordering issues, still.

Comment 58

[Jake Watkins [:dividehex]]

Comment on attachment 8868991 [details] [diff] [review]
Bug1315977_puppetv3.patch

Review of attachment 8868991 [details] [diff] [review]:
-----------------------------------------------------------------

First of all, the path attribute provides a search path for the command being executed.  In this case, /tools is not the location of 'rm' and the command it self provides an absolute path so 'path' can be omitted entirely. Second, 'creates' and 'onlyif'  both have the same job of prevent the exec from running.  You should only need one or the other.  Since you are already testing for the absents of '/tools/updates.txt' in the 'onlyif', the 'creates' is redundant.

I think the ordering is fine since there are ordering arrows separating ('->') all 3 resources although the anchoring in the middle makes it confusing.  In fact, I would move the anchoring completely outside the case for clarity and continuity.

As for the concern about this impacting non buildbot systems running 10.7, I don't think there are any and there certainly won't be any in the future.  I think that is a safe assumption.

If you are looking for a cleaner way of going about this, you might consider testing for the presence of the previous python installation.

exec { '/bin/rm -rf /tools/buildbot':
    onlyif => '/bin/test -f /var/db/.puppet_pkgdmg_installed_python27-2.7.3-1.dmg',
} ->
file { '/var/db/.puppet_pkgdmg_installed_python27-2.7.3-1.dmg':
    ensure => absent,
} ->
packages::pkgdmg {

...

Comment 59

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppetv4.patch

Created and tested new patch after the method that Jake recommended.
The test also look good,after upgrading python version and puppet is ran again /tools/buildbot is no longer deleted.
Some older tests that I've done in the past showed that buildbot can start and can connect to a dev-master after the puppet change.However would be wise first to make this change only for few machines(let's say 10 ) to see the status of more jobs before the change for all.
I can add that segment of machines if everything is fine so far.

Comment 60

[Kim Moir [:kmoir] ET]

Were you able to connect your newly configured machine to your dev-master and run a mac build on it?  This will should shake out any errors before moving to production.

Comment 61

[Jake Watkins [:dividehex]]

Comment on attachment 8872959 [details] [diff] [review]
bug1315977_puppetv4.patch

Review of attachment 8872959 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 62

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_subset.patch

(In reply to Kim Moir [:kmoir] from comment #60)
> Were you able to connect your newly configured machine to your dev-master
> and run a mac build on it?  This will should shake out any errors before
> moving to production.

I tried a different method of testing because the jobs will not be triggered if I try to connect the mac builds to my dev-master.
So I loaned 2 try build machines(bld-lion-r5-0[10,11]) and did some test on them,I manually ran puppet from my environment and the upgrade was made and  first job ran as expected (1) but after restart puppet was ran from production with the older version and the rest of jobs failed.
What I have in mind now is to create a subset with these 2 machines that will only ran my changes and will be pinned to my environment,the problem is that in the past when I did some tests,the jobs were not triggered and the machines remained idle if were pinned to my environment.Attached is this patch to create the subset with these 2 machines. 


(1)http://buildbot-master87.bb.releng.scl3.mozilla.com:8101/builders/OS%20X%2010.7%20try%20build/builds/34685

Comment 63

[Kim Moir [:kmoir] ET]

Comment on attachment 8874823 [details] [diff] [review]
bug1315977_subset.patch

Do you need modify to the range for the existing definition for mac build machines so they don't match both stanzas?

Comment 64

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Comment on attachment 8874823 [details] [diff] [review]
bug1315977_subset.patch

https://hg/build/puppet/rev/ca395d186dc9

Comment 65

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Did some tests on try mac and it seems that python version is changed back to 2.7.3 without needing ".puppet_pkgdmg_installed_python27-2.7.3-1.dmg" to exist.
I will need to re-think the puppet changes to always check for the newer python version ,because at this point,since ".puppet_pkgdmg_installed_python27-2.7.3-1.dmg" is never found we never check further if the new python version is installed.

Comment 66

[Amy Rich [:arr] [:arich]]

If we upgrade to 10.10 on the builders, does this problem just go away?

Comment 67

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

I don't think it will solve as versions '10.6','10.8','10.9','10.10' are set to use python 2.7.3,and for 10.7 we now test  python 2.7.12.
If we upgrade to 10.10 then the patch must be changed to make exception for 10.10 instead of 10.7,but this issue will still be found.

Comment 68

[Justin Wood (:Callek)]

Comment on attachment 8872959 [details] [diff] [review]
bug1315977_puppetv4.patch

Review of attachment 8872959 [details] [diff] [review]:
-----------------------------------------------------------------

Jake already saw this, I expect to have no comments that he wouldn't have noticed.

Comment 69

[Mihai Tabara [:mtabara]⌚️GMT]

I may be off tracks completely but given that we've moved to xcompiled builds on linux and we decommissioned (part of, I think) builders, do we still want this python lib upgrade to >= 2.7.6? Or the reason why we still want this is the ESR?

Comment 70

[Axel Hecht [:Pike]]

As of comment 48, I'm not the driver of this bug. I'd think this is releng-internal by now.

Comment 71

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Axel Hecht [:Pike] from comment #70)
> As of comment 48, I'm not the driver of this bug. I'd think this is
> releng-internal by now.

Okay, thanks for the update. Sorry for spam, skimmed the comments and missed that.
I'll redirect to :catlee since this is related to tcmigration.
@catlee - am I totally off the tracks in https://bugzilla.mozilla.org/show_bug.cgi?id=1315977#c69 or ESR makes us still want to proceed with this bug?

Comment 72

[Chris AtLee [:catlee]]

We'll still be using these build machines for ESR52, and for l10n repacks on beta/release for a while.

How much work is it to finish up?

Comment 73

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Chris AtLee [:catlee] from comment #72)
> We'll still be using these build machines for ESR52, and for l10n repacks on
> beta/release for a while.
> 
> How much work is it to finish up?

Hm, not sure. I'll discuss with :aobreja and will drop a message here.
For now, I'll drop a NI here for us both to keep in mind we need to come-up with something.

Comment 74

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

For the moment we don't have a simple solution to get rid of strings from the older version,the conditions in puppet from the last patch don't do a clean replace of python and rise new issues. 
Based on the #c50,we could have a possible workaround,delete /tools/buildbot every single time until changes are done on all machines and then change again the patch to only get the new version without deleting if we want this change very quick.
If this is not that urgent then I will continue testing for a cleaner solution.

Comment 75

[Mihai Tabara [:mtabara]⌚️GMT]

Bouncing this back to :catlee to take a decision with respect to solution/priority:

1. Should get go with the ugly solution short-term to get this done?
or
2. should we continue testing in puppet for a cleaner solution?

Comment 76

[Chris AtLee [:catlee]]

I think we should continue working on the method that Jake proposed in comment #58. We should understand why it's not working. Perhaps environment pinning isn't working between reboots?

Comment 77

[Jake Watkins [:dividehex]]

At this point it might be better to wipe and re-image the 10.7 builders to bypass all this hacky puppet logic we are attempting in order to deal with trying to clean up a previous python install.  In this case, version 2.7.3 never gets installed in the first place.  Starting fresh would probably be the cleanest and straight forward solution.

Comment 78

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

(In reply to Jake Watkins [:dividehex] from comment #77)
> At this point it might be better to wipe and re-image the 10.7 builders to
> bypass all this hacky puppet logic we are attempting in order to deal with
> trying to clean up a previous python install.  In this case, version 2.7.3
> never gets installed in the first place.  Starting fresh would probably be
> the cleanest and straight forward solution.

We can do what Jake suggested,we don't have many jobs running on bld-lion-r5 (under 10 per day):

https://www.hostedgraphite.com/da5c920d/86a8384e-d9cf-4208-989b-9538a1a53e4b/grafana/dashboard/db/pending?from=now-7d&to=now (filter after bld-lion-r5)

So for Builder and Try I can do tomorrow a downtime for 2 hours for all machines(around 80) and re-image them after pushing the patch to production.Once the first hosts will be re-imaged they will be enabled and can begin taking jobs.
The reason why we may still need a solution here is because in future we may want to upgrade python also for the test OS X (yosemite-r7) which use the same python version 2.7.3.
However if we don't want an upgrade for the OS X tests and everyone agree I can do tomorrow what Jake suggested,I will re-image 8 batches each with 10 machines.

Comment 79

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

For this change we can use attachment 8824421 [details] [diff] [review] from #c15 ,also we can consider using fewer machines since there are so few jobs.

Comment 80

[Chris AtLee [:catlee]]

We should hold off any downtime or upgrades until after next week I think, due to the upcoming release.

Comment 81

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Chris AtLee [:catlee] [PTO until Aug 14] from comment #80)
> We should hold off any downtime or upgrades until after next week I think,
> due to the upcoming release.

Yes. Any day after Wednesday should be good. But not until then.

Comment 82

[Mihai Tabara [:mtabara]⌚️GMT]

Note explaining the priority level: P2 doesn't mean we've lowered the priority, but the contrary. However, we're aligning these levels to the buildduty quarterly deliverables, where P1-P3 are taken by our daily waterline KTLO operational tasks.

This is something we want to look into this week. Or the next. Otherwise, we'd need to wait another release cycle to go past us to be able to make changes to the infrastructure.

Comment 83

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppet_split.patch

Added a patch to avoid a downtime and maybe to split the pool into 2 and upgrade each one at a time but I'm not able to make puppet recognize this fqdn,Jake do you have a better syntax for this example to include 014 and 015 in this case:

>      case $::fqdn {
>          /bld-lion-r5-01[4-5]\.try\.releng\.scl3\.mozilla\.com/: {
>              Anchor['packages::mozilla::python27::begin'] ->
>              packages::pkgdmg {

Comment 84

[Jake Watkins [:dividehex]]

Comment on attachment 8903196 [details] [diff] [review]
bug1315977_puppet_split.patch

Review of attachment 8903196 [details] [diff] [review]:
-----------------------------------------------------------------

::: manifests/moco-config.pp
@@ +5,3 @@
>  class config inherits config::base {
>      $org                   = 'moco'
> +    $puppet_notif_email    = '[e-mail]'

Make sure this doesn't land in prod.  We want to see the puppet emails also. :-)

::: modules/packages/manifests/mozilla/python27.pp
@@ +108,5 @@
> +                            }  -> Anchor['packages::mozilla::python27::end']
> +                        }
> +                        '10.7': {
> +                            case $::fqdn {
> +                                /bld-lion-r5-01[4-5]\.try\.releng\.scl3\.mozilla\.com/: {

Regex looks fine to me.  You could also try:

/^bld-lion-r5-01[45]\.try\.releng\.scl3\.mozilla\.com/

Or ...

/^bld-lion-r5-01(4|5)\.try\.releng\.scl3\.mozilla\.com/

Comment 85

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppet_v0.patch

This patch should upgrade after re-image python to version 2.7.12 for the bellow hosts:

bld-lion-r5-010.try.releng.scl3.mozilla.com
bld-lion-r5-011.try.releng.scl3.mozilla.com
bld-lion-r5-012.try.releng.scl3.mozilla.com
bld-lion-r5-013.try.releng.scl3.mozilla.com
bld-lion-r5-014.try.releng.scl3.mozilla.com
bld-lion-r5-015.try.releng.scl3.mozilla.com
bld-lion-r5-016.try.releng.scl3.mozilla.com

The change is only for these "try" machines so if anything will go wrong they can be easily disabled and the remaining 19 try machines which are not upgraded and are enabled can handle the load.

Comment 86

[Dragos Crisan [:dragrom]]

Comment on attachment 8903555 [details] [diff] [review]
bug1315977_puppet_v0.patch

Review of attachment 8903555 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me. Please correct the inline issues before land.

::: modules/packages/manifests/mozilla/python27.pp
@@ +102,5 @@
> +                    case $::macosx_productversion_major {
> +                        '10.6','10.8','10.9','10.10': {
> +                            Anchor['packages::mozilla::python27::begin'] ->
> +                            packages::pkgdmg {
> +                                python27:

'python27'

@@ +111,5 @@
> +                            case $::fqdn {
> +                                /^bld-lion-r5-01[^7-9]\.try\.releng\.scl3\.mozilla\.com/: {
> +                                    Anchor['packages::mozilla::python27::begin'] ->
> +                                    packages::pkgdmg {
> +                                        python27:

'python27'

@@ +113,5 @@
> +                                    Anchor['packages::mozilla::python27::begin'] ->
> +                                    packages::pkgdmg {
> +                                        python27:
> +                                            os_version_specific => true,
> +                                            version => "2.7.12-1";

arrange the arrow
use ' not "

@@ +119,5 @@
> +                                }
> +                                default: {
> +                                    Anchor['packages::mozilla::python27::begin'] ->
> +                                    packages::pkgdmg {
> +                                        python27:

'python27'

@@ +120,5 @@
> +                                default: {
> +                                    Anchor['packages::mozilla::python27::begin'] ->
> +                                    packages::pkgdmg {
> +                                        python27:
> +                                            version => "2.7.3-1";

use ' instead of "

Comment 87

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppet_v0.patch

Comment 88

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppet_v0.patch

Comment 89

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Comment on attachment 8903575 [details] [diff] [review]
bug1315977_puppet_v0.patch

https://hg.mozilla.org/build/puppet/rev/935cebce146d1f8404484aa7fd0b2bff6900b7f3

Comment 90

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Re-imaged all these machines and got the new python version installed without any strings from the older version:

>[[e-mail] ~]# /tools/buildbot/bin/python --version
>Python 2.7.12
>[[e-mail] ~]#

Comment 91

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppet_v1.patch

Patch to upgrade python to 2.7.12 for all try OS X machines and for build OS X range 060-079.

Comment 92

[Dragos Crisan [:dragrom]]

Comment on attachment 8904252 [details] [diff] [review]
bug1315977_puppet_v1.patch

LGTM

Comment 93

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Comment on attachment 8904252 [details] [diff] [review]
bug1315977_puppet_v1.patch

 https://hg.mozilla.org/build/puppet/rev/1f4ca4c5b1b802a8559cdb46197c6e16d6756068

Comment 94

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Re-imaged all these machines and enabled them with few exception(those that were loaned).
The exception list contain:

-bld-lion-r5-036.try.releng.scl3.mozilla.com
-bld-lion-r5-095.try.releng.scl3.mozilla.com
-bld-lion-r5-096.try.releng.scl3.mozilla.com

Comment 95

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppet_v2.patch

Everything went well with the previous changes so I add this final patch which should upgrade python to 2.7.12 for all 10.7 OS X.
In fact it will upgrade python to verion 2.7.12 for the remaining OS X build machines.

Comment 96

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1315977_puppet_v2.patch

Comment 97

[Dragos Crisan [:dragrom]]

Comment on attachment 8904583 [details] [diff] [review]
bug1315977_puppet_v2.patch

LGTM

Comment 98

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Comment on attachment 8904583 [details] [diff] [review]
bug1315977_puppet_v2.patch

https://hg.mozilla.org/build/puppet/rev/e2bff93ac850a6a24f33701f4b3d89d7de6a03e2

Comment 99

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Re-imaged remaining OS X build machines and re-enabled them in production.Everything seems to be done here.