629047 - Acceptance runs in buildbot should use PYTHON3 env var if available

Status: VERIFIED FIXED

(Tamarin Graveyard :: Build Config, defect, P4)

Description

[Brent Baker]

There has been some serious performance gains in how python handles threads in rev 3.2+ which can reduce acceptance run times on faster machines with many cores. Since it will not be possible to run the build slaves entirely on python3 (required build infrastructure is not expected to be fully ported for another year or so (Twisted, Zope, buildbot)) we should start running only the acceptance tests using python3. We already have a mechanism in place to execute the test/acceptance/runtests.py with a specified binary if the env variable exists (overriding the python in the path).

http://hg.mozilla.org/tamarin-redux/file/66974cb46b26/build/buildbot/slaves/all/run-acceptance-generic.sh#l122

This should be changed to use python3 specifically for running the acceptance suite.

a) PYTHONWIN -> PYTHON3
b) Update comment to reflect this change


MacBookPro (2 cores + hyperthreading) release acceptance
Py2.6 -> 92s
Py3.2 -> 59s

MacPro (12 cores + hyperthreading)
Py2.6 -> 32s
Py3.2 -> 16s

Comment 1

[Steven Johnson]

How backwards-compatible is Python3 with Python2.x? If it's pretty close we should consider upgrading all existing Python scripts and just requiring everyone to upgrade, for simplicity...

Comment 2

[Chris Peyer]

Unfortunately Py3 is not backwards compatible.  It is possible to update most scripts to be compatible in both 2 & 3 ... see Bug 627832. 

For tamarin I believe that we could update all of the scripts to be py2&3 compatible.  I'll open up another bug for that.

Comment 3

[Brent Baker]

Attachment: Run runtests.py with $PYTHON_RUNTESTS if available

Comment 4

[Chris Peyer]

Comment on attachment 508830 [details] [diff] [review]
Run runtests.py with $PYTHON_RUNTESTS if available

typo in this comment (s/to/,): # Not set to just use python that is in the path

I'm fine either way, but should we move the PYTHONWIN code into a function in environment.sh so it's not repeated in all the scripts?  The windows specific cygpath stuff could still be left in the diff scripts.

Comment 5

[Brent Baker]

Attachment: v2. run runtests.py using $PYTHON_RUNTESTS

- Centralized the check for the env variable in all/environment.sh
- If PYTHON_RUNTESTS is not set, then just use the python that is in the path

Comment 6

[Tamarin Bot]

changeset: 5897:70612842c6e7
user:      Brent Baker <[e-mail]>
summary:   Bug 629047: In buildbot run the testing framework using PYTHON_RUNTESTS env var if it is set, otherwise just use the python that is in the path (r=cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/70612842c6e7

Comment 7

[Tamarin Bot]

changeset: 5899:0ed235975c58
user:      Brent Baker <[e-mail]>
summary:   Bug 629047: missed one location that was still using  instead of /usr/local/bin/python3 (r=brbaker)

http://hg.mozilla.org/tamarin-redux/rev/0ed235975c58

Comment 8

[Tamarin Bot]

changeset: 5900:1bee76fb92ab
user:      Brent Baker <[e-mail]>
summary:   Bug 629047: one last tweak for how to run avmdiff on windows (r=brbaker)

http://hg.mozilla.org/tamarin-redux/rev/1bee76fb92ab