The default bug view has changed. See this FAQ.

activate fails to find 64-bit python on 64-bit windows environments

RESOLVED FIXED

Status

Add-on SDK
General
P2
normal
RESOLVED FIXED
7 years ago
3 years ago

People

(Reporter: glob, Assigned: markh)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [cherry-pick-1.3])

Attachments

(1 attachment, 9 obsolete attachments)

(Reporter)

Description

7 years ago
activate fails to find 64-bit python on 64-bit windows environments, as it assumes python is 32-bit.
(Reporter)

Updated

7 years ago
Version: unspecified → Trunk
(Reporter)

Comment 1

7 years ago
Created attachment 451013 [details]
activate.bat

updated activate.bat
Assignee: nobody → bugzilla
Created attachment 451023 [details] [diff] [review]
patch

patch out of an updated activate.bat
Attachment #451023 - Flags: review?(avarma)

Comment 3

7 years ago
Would it be more useful for Mossop to review this, since he wrote the original python-sniffing code?  My main concern is just that this doesn't break the current functionality of that batch file, i.e. that the batch file still successfully finds 32-bit python installs.
Comment on attachment 451023 [details] [diff] [review]
patch

(In reply to comment #3)
> Would it be more useful for Mossop to review this, since he wrote the original
> python-sniffing code?

Yes, good idea!
Attachment #451023 - Flags: review?(avarma) → review?(dtownsend)
Comment on attachment 451023 [details] [diff] [review]
patch

Not really sure I am the best to review this since I only really got the other working based on copy/paste/confusion/guesswork, but nonetheless something is certainly wrong here as it breaks finding python on my windows machine (64 bit windows 7 with 32 bit python)
Attachment #451023 - Flags: review?(dtownsend) → review-
(Reporter)

Comment 6

7 years ago
Created attachment 455345 [details] [diff] [review]
patch v2

fixes 32-bit python on 64-bit windows detection (sorry about that)
also fixes an error displayed on non-initial runs where the path contains a space
Attachment #451013 - Attachment is obsolete: true
Attachment #451023 - Attachment is obsolete: true
Attachment #455345 - Flags: review?(dtownsend)

Comment 7

7 years ago
Comment on attachment 455345 [details] [diff] [review]
patch v2

Brian, could you review this?
Attachment #455345 - Flags: review?(dtownsend) → review?(warner-bugzilla)
Wow, I'm afraid I haven't the faintest clue how to review this one: I've barely touched windows at all, and the little python-on-windows work I've done has never gotten as deep as this patch looks. Sorry!

Comment 9

7 years ago
Comment on attachment 455345 [details] [diff] [review]
patch v2

Atul, can you review this?  Who can review this?
Attachment #455345 - Flags: review?(warner-bugzilla) → review?(avarma)

Comment 10

7 years ago
Ugh, I can't review this either, unfortunately, as I'm very unfamiliar with Windows batch scripting (I mostly pasted together stuff from another framework's activate.bat when I made cfx's). Probably the best thing to do would be to get confirmation that this works with 32-bit python on 32-bit windows, at which point we know (I think) that all our bases are covered.

Updated

7 years ago
Attachment #455345 - Flags: review?(avarma)

Comment 11

7 years ago
Alternatively, we could just try committing it and see if anyone complains.

Comment 12

7 years ago
(In reply to comment #10)
> Probably the best thing to do
> would be to get confirmation that this works with 32-bit python on 32-bit
> windows, at which point we know (I think) that all our bases are covered.

But the point of this patch is to fix Win 7 64-bit isn't it?

How about this:  Ayan, could you apply this patch to the current SDK tip and test it on both a Windows 7 64-bit machine and a Windows 7 32-bit machine in the QA lab?
per irc, it would be great to QA this against other window 64 bit systems (XP first, then Vista)

Comment 14

7 years ago
It is, (In reply to comment #12)
> But the point of this patch is to fix Win 7 64-bit isn't it?

It is, but because no one is a Windows batch file or systems expert (and therefore able to review the code), the only thing I can think of doing to verify whether the code works is by actually running it on all known configurations of Windows. :(

Your suggestion for Ayan sounds great though!

Comment 15

7 years ago
Following up on Tony's comment 13 per IRC, people have reported potential 64-bit-related bin\activate.bat problems on Windows 7 [1, this bug], Vista [2], and possibly XP [3].  It's hard to tell whether all of these reports are correlated with 64-bit platforms or not.  But certainly since Byron marked this bug as occurring on Windows 7, we should test this patch on Windows 7 first.  If the patch doesn't work there, then Ayan, please attach a log of the failures.

I presume that bin\activate.bat fails on XP 64-bit and Vista 64-bit.  If that's the case and the patch works on Windows 7, then we should run the patch on XP and Vista to see if it works on those platforms too.

[1] http://groups.google.com/group/mozilla-labs-jetpack/browse_thread/thread/1670467cd304457b
[2] http://groups.google.com/group/mozilla-labs-jetpack/browse_thread/thread/f00be24389c68dea
[3] http://groups.google.com/group/mozilla-labs-jetpack/browse_thread/thread/cad1f4c2d9275e79

Comment 16

7 years ago
  Ayan, could you apply this patch to the current SDK tip and
> test it on both a Windows 7 64-bit machine and a Windows 7 32-bit machine in
> the QA lab?

I tested on windows vista(64bit) and windows xp(32 bit). Both failed to pass bin\activate. The results are as follows:

Win VISTA:-
C:\Users\marcia\Desktop\jetpack-sdk-31b3ae0dfd69\jetpack-sdk-31b3ae0dfd69>+PATH
C:\Program Files\Common Files\Microsoft Shared\Windows Live;C:\Windows\system32;
C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;
C:\Program Files (x86)\QuickTime\QTSystem\;C:\Python26\Scripts;C:\Python26;C:\Pr
ogram Files\Common Files\Microsoft Shared\Windows Live;
'+PATH' is not recognized as an internal or external command,
operable program or batch file.
$P$G was unexpected at this time.
C:\Users\marcia\Desktop\jetpack-sdk-31b3ae0dfd69\jetpack-sdk-31b3ae0dfd69> set P
ROMPT=(C:\Users\marcia\Desktop\jetpack-sdk-31b3ae0dfd69\jetpack-sdk-31b3ae0dfd69
) $P$G
C:\Users\marcia\Desktop\jetpack-sdk-31b3ae0dfd69\jetpack-sdk-31b3ae0dfd69>




Win XP:-

C:\Documents and Settings\mozilla\Desktop\jetpack-sdk\jetpack-sdk>+PATH C:\Progr
am Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files\PC Connecti
vity Solution\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Progra
m Files\QuickTime\QTSystem\;C:\Program Files\Common Files\Microsoft Shared\Windo
ws Live;C:\Python26\Scripts;C:\Python2;
'+PATH' is not recognized as an internal or external command,
operable program or batch file.
$P$G was unexpected at this time.
C:\Documents and Settings\mozilla\Desktop\jetpack-sdk\jetpack-sdk> set PROMPT=(C
:\Documents and Settings\mozilla\Desktop\jetpack-sdk\jetpack-sdk) $P$G
C:\Documents and Settings\mozilla\Desktop\jetpack-sdk\jetpack-sdk>

Comment 17

7 years ago
Created attachment 458827 [details]
activate.bat with patch v2 applied

Comment 18

7 years ago
Ayan and I took a look in the QA lab.  I don't know whether the patch helps or not.

Win7 64 fails with the current tip, without the patch:
  ERROR: The system was unable to find the specified registry key or value.
  \QuickTime\QTSystem\ was unexpected at this time.

Win7 64 fails with the patch applied:
  Warning: Failed to find Python installation directory
  Welcome to the Jetpack SDK.  Run 'cfx docs' for assistance.

Vista 64 works with the current tip, without the patch.

Vista 64 fails with the patch applied:
  Warning: Failed to find Python installation directory
  Welcome to the Jetpack SDK.  Run 'cfx docs' for assistance.

There wasn't an XP 64 machine handy.

The two failures with the patch are the same.  I don't know whether they're due to Python installation/config problems or a real problem with the patch.  I made sure the Python installations were included in the user's PATH, and that running `python` from the command line worked as expected.  The Win7 machine had python 2.7, the Vista machine 2.6.
(Reporter)

Comment 19

7 years ago
Created attachment 458891 [details] [diff] [review]
patch v3

on irc awd indicated that python 2.7 was installed on the win7-64 system, which the old version wouldn't detect.

this revision adds support for python 2.7, allows the batch file to be run from any directory, undoes an accident rollback of the welcome message, and fixes an issue with deactivate when the path contains a space.
Attachment #455345 - Attachment is obsolete: true
(Reporter)

Comment 20

7 years ago
Created attachment 458892 [details]
activate.bat with patch v3 applied
Attachment #458827 - Attachment is obsolete: true
(Reporter)

Comment 21

7 years ago
Created attachment 458894 [details]
activate.bat v3 with debugging messages

this version adds some debugging messages which should help diagnose failures.

here's the output from my win7-64 with python-32 2.6:

DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.7\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.6\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.5\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.4\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\InstallPath" /ve
ERROR: The system was unable to find the specified registry key or value.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath" /ve

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath
    (Default)    REG_SZ    C:\Python26\

Welcome to the Jetpack SDK. Run 'cfx docs' for assistance.

Comment 22

7 years ago
Hi,

I use your activate.bat v3 with debugging messages and get the following message when i try to excute bin\activate

DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.7\InstallPath" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.6\InstallPath" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.5\InstallPath" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Python\PythonCore\2.4\InstallPath" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\InstallPa
th" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPa
th" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.5\InstallPa
th" /ve
Das System kann den angegebenen Pfad nicht finden.
DEBUG: C:\Windows\sysnative\reg.exe QUERY "HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.4\InstallPa
th" /ve
Das System kann den angegebenen Pfad nicht finden.
Warning: Failed to find Python installation directory
** Welcome to the Jetpack SDK. Run 'cfx docs' for assistance

Python is correctly installed (2.7 - 64 Bit Version)

C:\Users\Boehrsi\Desktop\jetpack-sdk-0.5>python -V
Python 2.7

Python is installed in C:\Program Files\Python

mfg Boehrsi
(Reporter)

Comment 23

7 years ago
thanks boehrsi.  weird; python 2.7 64-bit works for me.

does the registry key HKLM\SOFTWARE\Python\PythonCore\2.7\InstallPath exist?

Comment 24

7 years ago
Yes it does with the correct path C:\Program Files\Python\
in HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\InstallPath

If i use the normal activate.bat (with a small change) i think python is located more or less correctly.

The change: 
if "%PYTHONVERSION%" EQU "" (
  REG QUERY "%PYTHONKEY%\2.7\InstallPath" /ve >nul 2>nul
  if %ERRORLEVEL% EQU 0 (
    SET PYTHONVERSION=2.7
  )
)

But if i test with cfx i get the following error.

Testing cfx...
Traceback (most recent call last):
  File "D:\jetpack-sdk-0.5\bin\cfx", line 6, in <module>
    cuddlefish.run()
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\__init__.py", line 318, in run
    test_all(env_root, defaults=options.__dict__)
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\__init__.py", line 190, in test_all
    result = test_cfx(env_root, defaults['verbose'])
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\__init__.py", line 215, in test_cfx
    retval = cuddlefish.tests.run(verbose)
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\tests\__init__.py", line 55, in run
    tests = get_tests()
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\tests\__init__.py", line 21, in get_tests
    module = __import__(full_name, fromlist=[package.__name__])
  File "D:\jetpack-sdk-0.5\python-lib\cuddlefish\runner.py", line 9, in <module>
    import mozrunner
  File "D:\jetpack-sdk-0.5\python-lib\mozrunner\__init__.py", line 48, in <module>
    import killableprocess
  File "D:\jetpack-sdk-0.5\python-lib\mozrunner\killableprocess.py", line 75, in <module>
    import winprocess
  File "D:\jetpack-sdk-0.5\python-lib\mozrunner\winprocess.py", line 39, in <module>
    from subprocess import SW_HIDE
ImportError: cannot import name SW_HIDE
(Reporter)

Comment 25

7 years ago
does C:\Windows\sysnative\reg.exe exist?
you'll have to check from the cmd prompt:

C:\>dir /b C:\Windows\sysnative\reg.exe
reg.exe

what version of windows are you running?

Comment 26

7 years ago
No this file does not exist, but the REG commmand is working fine.

I´m using
- Windows 7 Professional 64 Bit
- Python 2.7 64 Bit
(Reporter)

Comment 27

7 years ago
Created attachment 459339 [details]
activate.bat v4

ah, i think i know what's going on.  i use a hotkey launcher to load my cmd prompt, which happens to be a 32-bit app.  so i'm getting a 32-bit prompt, and a 32-bit reg.exe.

the 32-bit reg.exe can't find the 64-bit python without going via sysnative.

here's another activate.bat that fixes that.
Attachment #458891 - Attachment is obsolete: true
Attachment #458892 - Attachment is obsolete: true
Attachment #458894 - Attachment is obsolete: true
(Reporter)

Comment 28

7 years ago
i have no idea how to address the SW_HIDE error :(

winprocess.py does:
  from subprocess import SW_HIDE

but doesn't appear to reference it; deleting this line causes the error to vanish, but i don't know what side-effects this would cause.
(Reporter)

Comment 29

7 years ago
the SW_HIDE issue has been resolved in bug 581636

Comment 30

7 years ago
Hi all

It looks like this is already in hand, but a colleague and I have experience d this issue. Here's a link to a updated activate.bat that has fixed it for us. It also checks if Python is on the PATH first.

http://gist.github.com/644826

Kind regards

Graham
The Add-on SDK is no longer a Mozilla Labs experiment and has become a big enough project to warrant its own Bugzilla product, so the "Add-on SDK" product has been created for it, and I am moving its bugs to that product.

To filter bugmail related to this change, filter on the word "looptid".
Component: Jetpack SDK → General
Product: Mozilla Labs → Add-on SDK
QA Contact: jetpack-sdk → general
Version: Trunk → unspecified
Assignee: glob → myk
Priority: -- → P2
Target Milestone: --- → 1.0
(automatic reprioritization of 1.0 bugs)
Priority: P2 → P1
Target Milestone: 1.0 → 1.1
Priority: P1 → P2
Target Milestone: 1.1 → 1.2
(Assignee)

Comment 33

6 years ago
Was just helping someone in IRC which had this exact issue, so found this bug.  The version at https://gist.github.com/644826 is probably better, but still not ideal:

* It may find an unsupported version on PATH (eg, Python 3.x)

* It will not handle that you may have (say) a 32bit version of Python 3.x installed but a 64bit version of 2.6 installed - it will notice *some* 32bit Python is installed and from then on look only in the 32bit hive, therefore missing the 64bit 2.6.
(Assignee)

Comment 34

6 years ago
Sorry - I missed :glob's attachment - this looks good to me - I'll try to test it out soon.
(Pushing all open bugs to the --- milestone for the new triage system)
Target Milestone: 1.2 → ---
Assignee: myk → nobody
(Assignee)

Comment 36

6 years ago
Created attachment 567008 [details]
activate.bat markh tweaks

Attaching a tweaked version of glob's .bat file which works for me.  The main changes I made:

* verifies the python executables actually exist rather than trusting the registry.  Also handles the Python directory may have been built from source, so looks for the executables in a couple of places.

* works with windows XP - sadly XP's reg.exe has a different output format which made the parsing fail, so yet more "for" hackery is done to parse the output of the 'ver' command.

* Also looks in HKCU so it detects a Python installed "just for me".

* Cleans up the temp variables it sets.

I tested it on XP, Vista and Windows 7, and did the windows 7 tests with both 64bit and 32bit Pythons installed both "for all users" and "just for me".

Requesting review from Byron as the original author of the much-improved .bat file and also Wes as he just landed some HKCU support in bug 684136 - but feel free to decline to review if you have no time or inclination.
Assignee: nobody → mhammond
Attachment #459339 - Attachment is obsolete: true
Attachment #567008 - Flags: review?(kwierso)
Attachment #567008 - Flags: review?(glob)
Comment on attachment 567008 [details]
activate.bat markh tweaks

I'm not a reviewer, sorry. I'll try to play with this version of activate.bat in the morning and see how it works on my two Windows machines, but it's 3am at the moment.
Attachment #567008 - Flags: review?(kwierso)
(Assignee)

Comment 38

6 years ago
Comment on attachment 567008 [details]
activate.bat markh tweaks

Oops - as Wes points out, I'm after feedback :)
Attachment #567008 - Flags: review?(glob) → feedback?(glob)
(Assignee)

Updated

6 years ago
Attachment #567008 - Flags: feedback?(glob) → review?(myk)
(Assignee)

Comment 39

6 years ago
Created attachment 571798 [details] [diff] [review]
Minor fix

Oops - I attached a slightly old version - here is the good one.
Attachment #567008 - Attachment is obsolete: true
Attachment #571798 - Flags: review?(myk)
Attachment #567008 - Flags: review?(myk)
Comment on attachment 571798 [details] [diff] [review]
Minor fix

Looks good, r=myk

Note that the current file looks for Python 2.6 before Python 2.7, because that's what we tell people to use.  But what we actually test with is 2.5 on automated tests <https://tbpl.mozilla.org/?tree=Jetpack&usetinderbox=1> and presumably 2.7 in most manual testing, given that modern operating systems ship with 2.7 by default.

So I'm comfortable looking for 2.7 first (and in fact I think we should tell people to use it and test it automatically, but those are separate issues).
Attachment #571798 - Flags: review?(myk) → review+
https://github.com/mozilla/addon-sdk/commit/69ba50a3d51c877225174fca0b5a068f33cce6ff
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
https://github.com/mozilla/addon-sdk/commit/7f758d86996cc85bdaa021bd944cb1db9daa5ba7
Whiteboard: [cherry-pick-1.3]

Comment 43

3 years ago
I still have issues: 
- I have Windows 8.1 64bit
- My python 2.7 is installed in P:\Program Files (x86)\Python27\python.exe
- My SDK is intalled in P:\Program Files (x86)\Mozilla Firefox addin SDK\addon-sdk-1.17
- The reg key is in HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.7\InstallPath

I tried both the activate.bat from:
- the 1.17 sdk package, resulting in the script trying: set PYTHONINSTALL=P:\ProgramFiles(x86
- github, resulting in the script trying: SET "PYTHONINSTALL=P:\Program"

it should be:  SET "PYTHONINSTALL=P:\Program Files (x86)\Python27"
As you can see, the first cuts of at the bracket ) and removes the spaces, the second cuts of at the space
Flags: needinfo?(mhammond)
(Assignee)

Comment 44

3 years ago
You have the 32bit version of Python, not the 64bit version - but that isn't particularly relevant - the problem is going to be the fact Python is installed into a path with spaces.  This kind of problem is why Python itself suggests you install into "\Python" rather than into "\Program Files".

You can open a new bug for this and try to fix it if you like, but this bug isn't the correct place for further discussion on this specific issue.
Flags: needinfo?(mhammond)
You need to log in before you can comment on or make changes to this bug.