571843 - activate fails to find 64-bit python on 64-bit windows environments

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect, P2)

Description

[:glob ✱]

activate fails to find 64-bit python on 64-bit windows environments, as it assumes python is 32-bit.

Comment 1

[:glob ✱]

Attachment: activate.bat

updated activate.bat

Comment 2

[Zibi Braniecki [:zbraniecki][:gandalf]]

Attachment: patch

patch out of an updated activate.bat

Comment 3

[Atul Varma [:atul]]

Would it be more useful for Mossop to review this, since he wrote the original python-sniffing code?  My main concern is just that this doesn't break the current functionality of that batch file, i.e. that the batch file still successfully finds 32-bit python installs.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

Comment on attachment 451023 [details] [diff] [review]
patch

(In reply to comment #3)
> Would it be more useful for Mossop to review this, since he wrote the original
> python-sniffing code?

Yes, good idea!

Comment 5

[Dave Townsend [:mossop]]

Comment on attachment 451023 [details] [diff] [review]
patch

Not really sure I am the best to review this since I only really got the other working based on copy/paste/confusion/guesswork, but nonetheless something is certainly wrong here as it breaks finding python on my windows machine (64 bit windows 7 with 32 bit python)

Comment 6

[:glob ✱]

Attachment: patch v2

fixes 32-bit python on 64-bit windows detection (sorry about that)
also fixes an error displayed on non-initial runs where the path contains a space

Comment 7

[Drew Willcoxon :adw]

Comment on attachment 455345 [details] [diff] [review]
patch v2

Brian, could you review this?

Comment 8

[Brian Warner [:warner :bwarner]]

Wow, I'm afraid I haven't the faintest clue how to review this one: I've barely touched windows at all, and the little python-on-windows work I've done has never gotten as deep as this patch looks. Sorry!

Comment 9

[Drew Willcoxon :adw]

Comment on attachment 455345 [details] [diff] [review]
patch v2

Atul, can you review this?  Who can review this?

Comment 10

[Atul Varma [:atul]]

Ugh, I can't review this either, unfortunately, as I'm very unfamiliar with Windows batch scripting (I mostly pasted together stuff from another framework's activate.bat when I made cfx's). Probably the best thing to do would be to get confirmation that this works with 32-bit python on 32-bit windows, at which point we know (I think) that all our bases are covered.

Comment 11

[Atul Varma [:atul]]

Alternatively, we could just try committing it and see if anyone complains.

Comment 12

[Drew Willcoxon :adw]

(In reply to comment #10)
> Probably the best thing to do
> would be to get confirmation that this works with 32-bit python on 32-bit
> windows, at which point we know (I think) that all our bases are covered.

But the point of this patch is to fix Win 7 64-bit isn't it?

How about this:  Ayan, could you apply this patch to the current SDK tip and test it on both a Windows 7 64-bit machine and a Windows 7 32-bit machine in the QA lab?

Comment 13

[Tony Chung [:tchung]]

per irc, it would be great to QA this against other window 64 bit systems (XP first, then Vista)

Comment 14

[Atul Varma [:atul]]

It is, (In reply to comment #12)
> But the point of this patch is to fix Win 7 64-bit isn't it?

It is, but because no one is a Windows batch file or systems expert (and therefore able to review the code), the only thing I can think of doing to verify whether the code works is by actually running it on all known configurations of Windows. :(

Your suggestion for Ayan sounds great though!

Comment 15

[Drew Willcoxon :adw]

Following up on Tony's comment 13 per IRC, people have reported potential 64-bit-related bin\activate.bat problems on Windows 7 [1, this bug], Vista [2], and possibly XP [3].  It's hard to tell whether all of these reports are correlated with 64-bit platforms or not.  But certainly since Byron marked this bug as occurring on Windows 7, we should test this patch on Windows 7 first.  If the patch doesn't work there, then Ayan, please attach a log of the failures.

I presume that bin\activate.bat fails on XP 64-bit and Vista 64-bit.  If that's the case and the patch works on Windows 7, then we should run the patch on XP and Vista to see if it works on those platforms too.

[1] http://groups.google.com/group/mozilla-labs-jetpack/browse_thread/thread/1670467cd304457b
[2] http://groups.google.com/group/mozilla-labs-jetpack/browse_thread/thread/f00be24389c68dea
[3] http://groups.google.com/group/mozilla-labs-jetpack/browse_thread/thread/cad1f4c2d9275e79

Comment 16

[Ayan Shah[:ashah]]

  Ayan, could you apply this patch to the current SDK tip and
> test it on both a Windows 7 64-bit machine and a Windows 7 32-bit machine in
> the QA lab?

I tested on windows vista(64bit) and windows xp(32 bit). Both failed to pass bin\activate. The results are as follows:

Win VISTA:-
C:\Users\marcia\Desktop\jetpack-sdk-31b3ae0dfd69\jetpack-sdk-31b3ae0dfd69>+PATH
C:\Program Files\Common Files\Microsoft Shared\Windows Live;C:\Windows\system32;
C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;
C:\Program Files (x86)\QuickTime\QTSystem\;C:\Python26\Scripts;C:\Python26;C:\Pr
ogram Files\Common Files\Microsoft Shared\Windows Live;
'+PATH' is not recognized as an internal or external command,
operable program or batch file.
$P$G was unexpected at this time.
C:\Users\marcia\Desktop\jetpack-sdk-31b3ae0dfd69\jetpack-sdk-31b3ae0dfd69> set P
ROMPT=(C:\Users\marcia\Desktop\jetpack-sdk-31b3ae0dfd69\jetpack-sdk-31b3ae0dfd69
) $P$G
C:\Users\marcia\Desktop\jetpack-sdk-31b3ae0dfd69\jetpack-sdk-31b3ae0dfd69>




Win XP:-

C:\Documents and Settings\mozilla\Desktop\jetpack-sdk\jetpack-sdk>+PATH C:\Progr
am Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files\PC Connecti
vity Solution\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Progra
m Files\QuickTime\QTSystem\;C:\Program Files\Common Files\Microsoft Shared\Windo
ws Live;C:\Python26\Scripts;C:\Python2;
'+PATH' is not recognized as an internal or external command,
operable program or batch file.
$P$G was unexpected at this time.
C:\Documents and Settings\mozilla\Desktop\jetpack-sdk\jetpack-sdk> set PROMPT=(C
:\Documents and Settings\mozilla\Desktop\jetpack-sdk\jetpack-sdk) $P$G
C:\Documents and Settings\mozilla\Desktop\jetpack-sdk\jetpack-sdk>

Comment 17

[Drew Willcoxon :adw]

Attachment: activate.bat with patch v2 applied

Comment 18

[Drew Willcoxon :adw]

Ayan and I took a look in the QA lab.  I don't know whether the patch helps or not.

Win7 64 fails with the current tip, without the patch:
  ERROR: The system was unable to find the specified registry key or value.
  \QuickTime\QTSystem\ was unexpected at this time.

Win7 64 fails with the patch applied:
  Warning: Failed to find Python installation directory
  Welcome to the Jetpack SDK.  Run 'cfx docs' for assistance.

Vista 64 works with the current tip, without the patch.

Vista 64 fails with the patch applied:
  Warning: Failed to find Python installation directory
  Welcome to the Jetpack SDK.  Run 'cfx docs' for assistance.

There wasn't an XP 64 machine handy.

The two failures with the patch are the same.  I don't know whether they're due to Python installation/config problems or a real problem with the patch.  I made sure the Python installations were included in the user's PATH, and that running `python` from the command line worked as expected.  The Win7 machine had python 2.7, the Vista machine 2.6.

Comment 19

[:glob ✱]

Attachment: patch v3

on irc awd indicated that python 2.7 was installed on the win7-64 system, which the old version wouldn't detect.

this revision adds support for python 2.7, allows the batch file to be run from any directory, undoes an accident rollback of the welcome message, and fixes an issue with deactivate when the path contains a space.

Comment 20

[:glob ✱]

Attachment: activate.bat with patch v3 applied

Comment 21

[:glob ✱]

Attachment: activate.bat v3 with debugging messages

this version adds some debugging messages which should help diagnose failures.

here's the output from my win7-64 with python-32 2.6:

DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.7\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.6\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.5\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.4\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath" /ve

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath
    (Default)    REG_SZ    C:\Python26\

Welcome to the Jetpack SDK. Run 'cfx docs' for assistance.

Comment 22

[Boehrsi]

Hi,

I use your activate.bat v3 with debugging messages and get the following message when i try to excute bin\activate

DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.7\InstallPath" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.6\InstallPath" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.5\InstallPath" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.4\InstallPath" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\InstallPa
th" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPa
th" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.5\InstallPa
th" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.4\InstallPa
th" /ve
Das System kann den angegebenen Pfad nicht finden.
Warning: Failed to find Python installation directory
** Welcome to the Jetpack SDK. Run 'cfx docs' for assistance

Python is correctly installed (2.7 - 64 Bit Version)

C:\Users\Boehrsi\Desktop\jetpack-sdk-0.5>python -V
Python 2.7

Python is installed in C:\Program Files\Python

mfg Boehrsi

Comment 23

[:glob ✱]

thanks boehrsi.  weird; python 2.7 64-bit works for me.

does the registry key HKLM\SOFTWARE\Python\PythonCore\2.7\InstallPath exist?

Comment 24

[Boehrsi]

Yes it does with the correct path C:\Program Files\Python\
in HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\InstallPath

If i use the normal activate.bat (with a small change) i think python is located more or less correctly.

The change: 
if "%PYTHONVERSION%" EQU "" (
  REG QUERY "%PYTHONKEY%\2.7\InstallPath" /ve >nul 2>nul
  if %ERRORLEVEL% EQU 0 (
    SET PYTHONVERSION=2.7
  )
)

But if i test with cfx i get the following error.

Testing cfx...
Traceback (most recent call last):
  File "D:\jetpack-sdk-0.5\bin\cfx", line 6, in <module>
    cuddlefish.run()
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\__init__.py", line 318, in run
    test_all(env_root, defaults=options.__dict__)
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\__init__.py", line 190, in test_all
    result = test_cfx(env_root, defaults['verbose'])
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\__init__.py", line 215, in test_cfx
    retval = cuddlefish.tests.run(verbose)
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\tests\__init__.py", line 55, in run
    tests = get_tests()
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\tests\__init__.py", line 21, in get_tests
    module = __import__(full_name, fromlist=[package.__name__])
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\runner.py", line 9, in <module>
    import mozrunner
  File "D:\jetpack-sdk-0.5\python-lib\mozrunner\__init__.py", line 48, in <module>
    import killableprocess
  File "D:\jetpack-sdk-0.5\python-lib\mozrunner\killableprocess.py", line 75, in <module>
    import winprocess
  File "D:\jetpack-sdk-0.5\python-lib\mozrunner\winprocess.py", line 39, in <module>
    from subprocess import SW_HIDE
ImportError: cannot import name SW_HIDE

Comment 25

[:glob ✱]

does C:\Windows\sysnative\reg.exe exist?
you'll have to check from the cmd prompt:

C:\>dir /b C:\Windows\sysnative\reg.exe
reg.exe

what version of windows are you running?

Comment 26

[Boehrsi]

No this file does not exist, but the REG commmand is working fine.

I´m using
- Windows 7 Professional 64 Bit
- Python 2.7 64 Bit

Comment 27

[:glob ✱]

Attachment: activate.bat v4

ah, i think i know what's going on.  i use a hotkey launcher to load my cmd prompt, which happens to be a 32-bit app.  so i'm getting a 32-bit prompt, and a 32-bit reg.exe.

the 32-bit reg.exe can't find the 64-bit python without going via sysnative.

here's another activate.bat that fixes that.

Comment 28

[:glob ✱]

i have no idea how to address the SW_HIDE error :(

winprocess.py does:
  from subprocess import SW_HIDE

but doesn't appear to reference it; deleting this line causes the error to vanish, but i don't know what side-effects this would cause.

Comment 29

[:glob ✱]

the SW_HIDE issue has been resolved in bug 581636

Comment 30

[Graham Scott]

Hi all

It looks like this is already in hand, but a colleague and I have experience d this issue. Here's a link to a updated activate.bat that has fixed it for us. It also checks if Python is on the PATH first.

http://gist.github.com/644826

Kind regards

Graham

Comment 31

[Myk Melez [:myk] [@mykmelez]]

The Add-on SDK is no longer a Mozilla Labs experiment and has become a big enough project to warrant its own Bugzilla product, so the "Add-on SDK" product has been created for it, and I am moving its bugs to that product.

To filter bugmail related to this change, filter on the word "looptid".

Comment 32

[Myk Melez [:myk] [@mykmelez]]

(automatic reprioritization of 1.0 bugs)

Comment 33

[Mark Hammond [:markh] [:mhammond]]

Was just helping someone in IRC which had this exact issue, so found this bug.  The version at https://gist.github.com/644826 is probably better, but still not ideal:

* It may find an unsupported version on PATH (eg, Python 3.x)

* It will not handle that you may have (say) a 32bit version of Python 3.x installed but a 64bit version of 2.6 installed - it will notice *some* 32bit Python is installed and from then on look only in the 32bit hive, therefore missing the 64bit 2.6.

Comment 34

[Mark Hammond [:markh] [:mhammond]]

Sorry - I missed :glob's attachment - this looks good to me - I'll try to test it out soon.

Comment 35

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(Pushing all open bugs to the --- milestone for the new triage system)

Comment 36

[Mark Hammond [:markh] [:mhammond]]

Attachment: activate.bat markh tweaks

Attaching a tweaked version of glob's .bat file which works for me.  The main changes I made:

* verifies the python executables actually exist rather than trusting the registry.  Also handles the Python directory may have been built from source, so looks for the executables in a couple of places.

* works with windows XP - sadly XP's reg.exe has a different output format which made the parsing fail, so yet more "for" hackery is done to parse the output of the 'ver' command.

* Also looks in HKCU so it detects a Python installed "just for me".

* Cleans up the temp variables it sets.

I tested it on XP, Vista and Windows 7, and did the windows 7 tests with both 64bit and 32bit Pythons installed both "for all users" and "just for me".

Requesting review from Byron as the original author of the much-improved .bat file and also Wes as he just landed some HKCU support in bug 684136 - but feel free to decline to review if you have no time or inclination.

Comment 37

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Comment on attachment 567008 [details]
activate.bat markh tweaks

I'm not a reviewer, sorry. I'll try to play with this version of activate.bat in the morning and see how it works on my two Windows machines, but it's 3am at the moment.

Comment 38

[Mark Hammond [:markh] [:mhammond]]

Comment on attachment 567008 [details]
activate.bat markh tweaks

Oops - as Wes points out, I'm after feedback :)

Comment 39

[Mark Hammond [:markh] [:mhammond]]

Attachment: Minor fix

Oops - I attached a slightly old version - here is the good one.

Comment 40

[Myk Melez [:myk] [@mykmelez]]

Comment on attachment 571798 [details] [diff] [review]
Minor fix

Looks good, r=myk

Note that the current file looks for Python 2.6 before Python 2.7, because that's what we tell people to use.  But what we actually test with is 2.5 on automated tests <https://tbpl.mozilla.org/?tree=Jetpack&usetinderbox=1> and presumably 2.7 in most manual testing, given that modern operating systems ship with 2.7 by default.

So I'm comfortable looking for 2.7 first (and in fact I think we should tell people to use it and test it automatically, but those are separate issues).

Comment 41

[Myk Melez [:myk] [@mykmelez]]

https://github.com/mozilla/addon-sdk/commit/69ba50a3d51c877225174fca0b5a068f33cce6ff

Comment 42

[Myk Melez [:myk] [@mykmelez]]

https://github.com/mozilla/addon-sdk/commit/7f758d86996cc85bdaa021bd944cb1db9daa5ba7

Comment 43

[coolcat_the_best2]

I still have issues: 
- I have Windows 8.1 64bit
- My python 2.7 is installed in P:\Program Files (x86)\Python27\python.exe
- My SDK is intalled in P:\Program Files (x86)\Mozilla Firefox addin SDK\addon-sdk-1.17
- The reg key is in HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\InstallPath

I tried both the activate.bat from:
- the 1.17 sdk package, resulting in the script trying: set PYTHONINSTALL=P:\ProgramFiles(x86
- github, resulting in the script trying: SET "PYTHONINSTALL=P:\Program"

it should be:  SET "PYTHONINSTALL=P:\Program Files (x86)\Python27"
As you can see, the first cuts of at the bracket ) and removes the spaces, the second cuts of at the space

Comment 44

[Mark Hammond [:markh] [:mhammond]]

You have the 32bit version of Python, not the 64bit version - but that isn't particularly relevant - the problem is going to be the fact Python is installed into a path with spaces.  This kind of problem is why Python itself suggests you install into "\Python" rather than into "\Program Files".

You can open a new bug for this and try to fix it if you like, but this bug isn't the correct place for further discussion on this specific issue.