Closed Bug 673132 Opened 13 years ago Closed 12 years ago

update talos python requirement from python 2.4 to python 2.5

Categories

(Testing :: Talos, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jmaher, Unassigned)

References

()

Details

(Whiteboard: [talos])

Attachments

(2 files, 2 obsolete files)

currently talos required python 2.4 and our unittests (which are run on the same machines) require python 2.5.  We should keep our talos requirements the same as our unittest requirements.

One big thing to resolve is the windows perf libraries (for counter collection) that are installed in the python2.4 installation.  This might be as easy as installing those libraries in a python2.5 installation, but I suspect there might be a little more.  Also these changes that we discover in staging would need to be rolled out to the talos slaves (by releng).
Whiteboard: [talos]
the buildbot code might not be too hard to update:
http://mxr.mozilla.org/build/source/buildbotcustom/env.py#136
Also, http://sourceforge.net/projects/pywin32/files/pywin32/Build216/

Are these easy_installable on windows?  IMHO, it would make the most sense to make talos a setuptools/distribute package and peg to pywin32 if applicable or similar, though I don't know the details of the deployment story (only enough to be afraid).
What is our way forward here?  I think it's this:

Once we know that talos works fine in our staging environment with 2.5 + dependencies we need to then file a releng bug (or morph this into one) for the puppet/opsi work to update the slaves.
I agree with :ctalbert here, lets just figure out the dependencies and how to install them properly in an automated fashion and we can get it scheduled in over the next few weeks.

In our ideal solution it would use virtualenv, etc...  Unfortunately we are not ready for that and moving forward would be nice.
OS: Linux → All
Hardware: x86 → All
This adds the yaml requirement to setup.py. pywin32 will be more difficult :(
Attachment #570735 - Flags: review?(jmaher)
(In reply to Joel Maher (:jmaher) from comment #4)
> it appears so:
> easy_install
> http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/
> pywin32-216.win32-py2.6.exe
> 
> * taken from: https://wiki.mozilla.org/StandaloneTalos#Prerequisites *

I updated the instructions there.  First, to note, the URL is python 2.6 specific.  I'll play around with this in a bit on windows to see what we're supposed to do for windows, but it looks like this may be more complicated.
output of my easy_install:

(talos)│easy_install http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win32-py2.7.exe
Downloading http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win32-py2.7.exe
Processing pywin32-216.win32-py2.7.exe
win32timezone: module references __file__
win32evtlogutil: module references __file__
win32serviceutil: module references __file__
pywintypes: module references __file__
test.test_clipboard: module references __file__
test.test_win32rcparser: module references __file__
test.testall: module references __file__
test.test_win32api: module references __file__
/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/bdist_egg.py:422: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
test.test_win32trace: module references __file__
isapi.install: module references __file__
isapi.test.setup: module references __file__
isapi.samples.test: module references __file__
isapi.samples.advanced: module references __file__
pywin.framework.startup: module references __path__
pywin.framework.scriptutils: module references __file__
pywin.scintilla.control: module references __file__
pywin.scintilla.IDLEenvironment: module references __file__
pywin.scintilla.config: module references __path__
win32com.__init__: module references __path__
win32com.test.testDictionary: module references __file__
win32com.test.pippo_server: module references __file__
win32com.test.GenTestScripts: module references __path__
win32com.test.testall: module references __file__
win32com.test.testAXScript: module references __path__
win32com.test.testPyComTest: module references __file__
win32com.test.testPippo: module references __file__
win32com.test.util: module references __file__
win32com.test.testServers: module references __file__
win32com.server.register: module references __file__
win32com.server.register: module references __path__
scripts.regsetup: module references __file__
scripts.setup_d: module references __file__
Demos.win32rcparser_demo: module references __file__
Demos.security.sspi.validate_password: module references __file__
win32comext.bits.__init__: module references __path__
win32comext.shell.__init__: module references __path__
win32comext.shell.demos.servers.folder_view: module references __file__
win32comext.shell.demos.servers.empty_volume_cache: module references __file__
win32comext.axcontrol.__init__: module references __path__
win32comext.axscript.__init__: module references __path__
win32comext.axdebug.__init__: module references __path__
win32comext.axdebug.debugger: module references __file__
win32comext.axdebug.codecontainer: module references __file__
win32comext.internet.__init__: module references __path__
win32comext.authorization.__init__: module references __path__
win32comext.directsound.__init__: module references __path__
win32comext.directsound.test.ds_test: module references __file__
win32comext.mapi.__init__: module references __path__
win32comext.taskscheduler.__init__: module references __path__
win32comext.adsi.__init__: module references __path__
creating '/tmp/easy_install-cAx0Qh/pywin32-216-py2.7-win32.egg' and adding '/tmp/easy_install-cAx0Qh/pywin32-216-py2.7-win32.egg.tmp' to it
removing '/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/pywin32-216-py2.7-win32.egg' (and everything under it)
creating /home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/pywin32-216-py2.7-win32.egg
Extracting pywin32-216-py2.7-win32.egg to /home/jhammel/mozilla/src/talos/lib/python2.7/site-packages
pywin32 216 is already the active version in easy-install.pth
Installing pywin32_postinstall.py script to /home/jhammel/mozilla/src/talos/bin
Installing pywin32_testall.py script to /home/jhammel/mozilla/src/talos/bin
Installing pywin32_testall.pyc script to /home/jhammel/mozilla/src/talos/bin
Installing pywin32_postinstall.pyc script to /home/jhammel/mozilla/src/talos/bin

Installed /home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/pywin32-216-py2.7-win32.egg
Processing dependencies for pywin32==216
Searching for pywin32==216
Reading http://pypi.python.org/simple/pywin32/
Reading http://sf.net/projects/pywin32
Reading https://sourceforge.net/projects/pywin32/files/pywin32/
Reading http://sourceforge.net/project/showfiles.php?group_id=78018
No local packages or download links found for pywin32==216
Best match: None
Traceback (most recent call last):
  File "/home/jhammel/mozilla/src/talos/bin/easy_install", line 8, in <module>
    load_entry_point('setuptools==0.6c11', 'console_scripts', 'easy_install')()
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 211, in run
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 422, in easy_install
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 478, in install_item
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 519, in process_distribution
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 563, in resolve
    plugin_env, full_env=None, installer=None, fallback=True
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 799, in best_match
    
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 811, in obtain
    self.add(other)
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 434, in easy_install
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/package_index.py", line 475, in fetch_distribution
AttributeError: 'NoneType' object has no attribute 'clone'
This may not be useful information, but I remember when I last set up talos on Windows I wound up installing the pywin32 extensions manually via the .exe file because of this exact issue. There may be a solution here but I'm not sure what it is.
Comment on attachment 570735 [details] [diff] [review]
add pyyaml requirement

Looks good to me.

(reviewing on request from Jeff)
Attachment #570735 - Flags: review?(jmaher) → review+
(In reply to William Lachance (:wlach) from comment #11)
> Comment on attachment 570735 [details] [diff] [review] [diff] [details] [review]
> add pyyaml requirement
> 
> Looks good to me.
> 
> (reviewing on request from Jeff)

pushed: http://hg.mozilla.org/build/talos/rev/0d5f418718fa
If you give all the direct download links and tell setuptools about the dependency, it actually finds the correct one according to the platform, downloads and installs properly.

I have also thought about creating a simple Google App Engine service which will take http://sourceforge.net/projects/pywin32/files/pywin32/Build216/ or similar links and generate a new file including the direct links to the files listed there but saw little gain from that project and abandoned the idea. If you think that is a better solution, please tell it.
Attachment #580774 - Flags: review?(wlachance)
Attachment #580774 - Flags: review?(jhammel)
Comment on attachment 580774 [details] [diff] [review]
Added pywin32 requirement with proper link generation

-dependencies = ['PyYAML']
+dependencies = ['PyYAML', 'pywin32']

We should only add the pywin32 dependency on the windows platform.  I'm also guessing we should only add the dependency link for the version that we want. (IIRC, you also may also need a #pywin32 hash anchor for the dependency link URL).

+def generate_sf_links(base,
+                      platforms=['win32', 'win-amd64'],
+                      versions=['2.5', '2.6', '2.7', '3.1', '3.2']):

Since the defaults are immutable, they should be tuples (e.g. `platforms=('win32', 'win-amd64')`).  See http://k0s.org/hg/config/file/tip/python/mutable_defaults.py

Its close, but not quite there yet
Attachment #580774 - Flags: review?(jhammel) → review-
(In reply to Jeff Hammel [:jhammel] from comment #14)
> We should only add the pywin32 dependency on the windows platform.  I'm also
> guessing we should only add the dependency link for the version that we
> want.
setuptools actually chooses the correct one according to the platform so I though listing all possible links and leaving choosing the correct one according to the platform was a better idea. What do you think?

>(IIRC, you also may also need a #pywin32 hash anchor for the dependency link URL).
It currently works with the url pattern I provided so I'm not sure why this is needed. Can you give more information?

Btw. thanks for the review, will fix ASAP! =)
(In reply to Burak Yiğit Kaya from comment #15)
> (In reply to Jeff Hammel [:jhammel] from comment #14)
> > We should only add the pywin32 dependency on the windows platform.  I'm also
> > guessing we should only add the dependency link for the version that we
> > want.
> setuptools actually chooses the correct one according to the platform so I
> though listing all possible links and leaving choosing the correct one
> according to the platform was a better idea. What do you think?

I'm a bit vague on the innards, but certainly pywin32 cannot be installed on linux:

(talos)│curl https://bug673132.bugzilla.mozilla.org/attachment.cgi?id=580774 | patch -p1 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   998  100   998    0     0   2035      0 --:--:-- --:--:-- --:--:--  3024
patching file setup.py
(talos)│python setup.py develop
running develop
running egg_info
writing requirements to talos.egg-info/requires.txt
writing talos.egg-info/PKG-INFO
writing top-level names to talos.egg-info/top_level.txt
writing dependency_links to talos.egg-info/dependency_links.txt
writing entry points to talos.egg-info/entry_points.txt
reading manifest file 'talos.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'talos.egg-info/SOURCES.txt'
running build_ext
Creating /home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/talos.egg-link (link to .)
talos 0.0 is already the active version in easy-install.pth

Installed /home/jhammel/mozilla/src/talos/src/talos
Processing dependencies for talos==0.0
Searching for pywin32
Reading http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win-amd64-py2.5.exe
Reading http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win-amd64-py2.6.exe
Reading http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win-amd64-py2.7.exe
Reading http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win-amd64-py3.1.exe
Reading http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win-amd64-py3.2.exe
Reading http://pypi.python.org/simple/pywin32/
Reading http://sf.net/projects/pywin32
Reading http://sourceforge.net/project/showfiles.php?group_id=78018
Reading https://sourceforge.net/projects/pywin32/files/pywin32/
No local packages or download links found for pywin32
Best match: None
Traceback (most recent call last):
  File "setup.py", line 58, in <module>
    """,
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/develop.py", line 27, in run
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/develop.py", line 102, in install_for_development
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 519, in process_distribution
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 563, in resolve
    plugin_env, full_env=None, installer=None, fallback=True
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 799, in best_match
    
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 811, in obtain
    self.add(other)
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 434, in easy_install
  File "/home/jhammel/mozilla/src/talos/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/package_index.py", line 475, in fetch_distribution
AttributeError: 'NoneType' object has no attribute 'clone'

Being vague on the details I don't really have a strong preference as far as having setuptools figure out what its going to install, but I'd rather err on the side of being more precise than not precise enough.  As long as it works
 
> >(IIRC, you also may also need a #pywin32 hash anchor for the dependency link URL).
> It currently works with the url pattern I provided so I'm not sure why this
> is needed. Can you give more information?

I've always been a bit vague on the details so I've cargo-culted and put in the egg_info hash tag.  There is a bit here: http://stackoverflow.com/questions/3472430/how-can-i-make-setuptools-install-a-package-thats-not-on-pypi ; and here: http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-in-pypi Though TBH its hard to tell what is what. setuptools docs are infamous for being vague at best (and outright wrong in a few cases).

> 
> Btw. thanks for the review, will fix ASAP! =)

Thanks for looking into this!
Fixed the issues :jhammel mentioned.
Attachment #580774 - Attachment is obsolete: true
Attachment #580774 - Flags: review?(wlachance)
Attachment #580783 - Flags: review?(jhammel)
(In reply to Jeff Hammel [:jhammel] from comment #16)
> Being vague on the details I don't really have a strong preference as far as
> having setuptools figure out what its going to install, but I'd rather err
> on the side of being more precise than not precise enough.  As long as it
> works
I made it to determine the platform though I had to use "sys.maxsize" to determine the Python platform(win32 or amd64) which seems reliable enough though I didn't like much.

> > >(IIRC, you also may also need a #pywin32 hash anchor for the dependency link URL).
> > It currently works with the url pattern I provided so I'm not sure why this
> > is needed. Can you give more information?
> I've always been a bit vague on the details so I've cargo-culted and put in
> the egg_info hash tag.  There is a bit here:
> http://stackoverflow.com/questions/3472430/how-can-i-make-setuptools-install-
> a-package-thats-not-on-pypi ; and here:
> http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-
> in-pypi Though TBH its hard to tell what is what. setuptools docs are
> infamous for being vague at best (and outright wrong in a few cases).
It says: 'If you depend on a package that's distributed as a single .py file, you must include an "#egg=project-version" suffix to the URL, to give a project name and version number. ... EasyInstall will recognize this suffix and automatically create a trivial setup.py to wrap the single .py file as an egg.' which I think is not the case for pywin32 and I also checked it without the hash and it worked so added the patch without it =)

> > Btw. thanks for the review, will fix ASAP! =)
> Thanks for looking into this!
It's a pleasure!
Comment on attachment 580783 [details] [diff] [review]
Added pywin32 requirement with proper link generation for Windows platforms

+    platform_name = 'win-amd64' if is_64bits else 'win32'

Sadly, this construct is a python 2.5ism and talos is still required to support 2.4 in production currently (yes, I realize the title of the bug would make you think this is all that's required, but....).  You can do:

is_64bits and 'win-amd64' or 'win32'

or the more canonical

if is_64bits:
   platform_name = 'win-amd64'
else:
   platform_name == 'win32'

r+ with this change
Attachment #580783 - Flags: review?(jhammel) → review+
(In reply to Burak Yiğit Kaya from comment #18)
> (In reply to Jeff Hammel [:jhammel] from comment #16)
> > Being vague on the details I don't really have a strong preference as far as
> > having setuptools figure out what its going to install, but I'd rather err
> > on the side of being more precise than not precise enough.  As long as it
> > works
> I made it to determine the platform though I had to use "sys.maxsize" to
> determine the Python platform(win32 or amd64) which seems reliable enough
> though I didn't like much.

I *think* this is okay?  See also how we do it in mozinfo: https://github.com/mozilla/mozbase/blob/master/mozinfo/mozinfo/mozinfo.py  This isn't necessarily the canonical way, but it seems to work on all platforms tested on.

> > > >(IIRC, you also may also need a #pywin32 hash anchor for the dependency link URL).
> > > It currently works with the url pattern I provided so I'm not sure why this
> > > is needed. Can you give more information?
> > I've always been a bit vague on the details so I've cargo-culted and put in
> > the egg_info hash tag.  There is a bit here:
> > http://stackoverflow.com/questions/3472430/how-can-i-make-setuptools-install-
> > a-package-thats-not-on-pypi ; and here:
> > http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-
> > in-pypi Though TBH its hard to tell what is what. setuptools docs are
> > infamous for being vague at best (and outright wrong in a few cases).
> It says: 'If you depend on a package that's distributed as a single .py
> file, you must include an "#egg=project-version" suffix to the URL, to give
> a project name and version number. ... EasyInstall will recognize this
> suffix and automatically create a trivial setup.py to wrap the single .py
> file as an egg.' which I think is not the case for pywin32 and I also
> checked it without the hash and it worked so added the patch without it =)

As said, I don't necessarily trust the setuptools docs ;)  I can say that in the past I have not put egg= hashes in, and not for single .py files, and have been bitten. (FWIW, the case mentioned on stackoverflow is not a single python file). Maybe this information is determined from the URL?  :shrug: In any case, I am fine without it, as long as it works, and if it doesn't hopefully someone can report it and it would be easy to fix.

> > > Btw. thanks for the review, will fix ASAP! =)
> > Thanks for looking into this!
> It's a pleasure!
Added Python 2.4 support. Readability > micro-optimization, especially in Python ;)
Attachment #580783 - Attachment is obsolete: true
Attachment #580834 - Flags: review?(jhammel)
Comment on attachment 580834 [details] [diff] [review]
Added pywin32 requirement with proper link generation for Windows platforms w/ py 2.4 support

Awesome :)
Attachment #580834 - Flags: review?(jhammel) → review+
(In reply to Jeff Hammel [:jhammel] from comment #22)
> Comment on attachment 580834 [details] [diff] [review]
> Added pywin32 requirement with proper link generation for Windows platforms
> w/ py 2.4 support
> 
> Awesome :)

I've tested this on linux via the test_install.py script as well as just running `python setup.py develop`. It worked fine.  I also ran through staging.  Will test 'ts' on my tablet next (though, again, the setup.py code isn't touched).  If that works, someone should test on windows, but i think we're safe to land
Now tested with a tablet
Tested extensively on windows XP as well as other flavors.  Ready to push :)  Thanks, ckvk!
(In reply to Jeff Hammel [:jhammel] from comment #22)
> Comment on attachment 580834 [details] [diff] [review]
> Added pywin32 requirement with proper link generation for Windows platforms
> w/ py 2.4 support
> 
> Awesome :)

pushed: http://hg.mozilla.org/build/talos/rev/cb1b7a64f98e
If we are pursuing the path of getting talos installed into a virtualenv on production talos slaves at least desktop talos harness and an internal pypi server are needed.  In addition, the way buildbot is done will have to be completely rejiggered.  I'll put those two things as blocking bugs and we probably want to add more
Blocks: 724191
Depends on: 726700
Depends on: 734136
After some discussion, we decided that it's not really worth the effort of upgrading talos to use python 2.5 was not worth the hassle. Every time we make a change like this, we need to do some in-depth testing to make sure the numbers won't change too much. If we're going to do that, may as well bite the bullet and move to python 2.7.

The work to do this is being tracked in bug 734466
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
Summary: update talos python requirement from python 2.4 to python 2.5 → update talos python requirement from python 2.4 to python 2.7
Accidentally changed title of bug when resolving
Summary: update talos python requirement from python 2.4 to python 2.7 → update talos python requirement from python 2.4 to python 2.5
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: