1344643 - Use 64bits Python in MozillaBuild

Status: RESOLVED FIXED

(Firefox Build System :: MozillaBuild, task)

Description

[Jerry Shih[:jerry] (UTC+8) (inactive)]

I hit an OOM problem when I use git-cinnabar to clone the mozilla-central code-base on windows platform. The 64bits python seems workable.
Please check the following issue:
https://github.com/glandium/git-cinnabar/issues/110

Could we turn to use 64bits python in mozillaBuild? Or could we have one more 64bits version of mozillaBuild?

Comment 1

[Ryan VanderMeulen [:RyanVM]]

MozillaBuild is basically on life support at this point due to the long-promised (and at this point, long-overdue) transition to msys2 instead. However, yours isn't the only recent complaint I've heard about 32-bit python OOM issues, so I'm seriously considering doing another MozillaBuild release at this point and making a 64-bit version to go with it.

No promises, but thanks for the ping.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Here's a version of MozillaBuild that includes 64-bit Python and hg. No warranties, but if you want to try it out, let me know how it treats you :)

https://people-mozilla.org/~rvandermeulen/MozillaBuild64Setup2.3.0pre.exe
sha256: ac2ec6cf873e67eb316be17965822368eb3e43dbcae7dfa68a71ae1d4d95cb15

Comment 3

[Jerry Shih[:jerry] (UTC+8) (inactive)]

I don't see the oom problem with git-cinnabar!
And I tried the "mach build". It also works.
I will try the mozreview later.
Thanks, Ryan.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Per https://groups.google.com/d/msg/mozilla.dev.builds/MbBQzIpWX_c/ca2zIojoCAAJ, I intend to make MozillaBuild 2.3 require a 64bit host OS (I'm pretty sure we've long been unable to build Firefox on 32bit systems due to MSVC OOMing), so we can just make this switch as part of that.

Comment 5

[Kelsey Gilbert [:jgilbert]]

Big problem here, at least with vs2017:
From build/moz.configure/windows.configure:
list(get_registry_values(
        r'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Kits\Installed Roots'
        r'\KitsRoot*')):
mozbuild2.2:
[('KitsRoot10', u'C:\\Program Files (x86)\\Windows Kits\\10\\'), ('KitsRoot81', u'C:\\Program Files (x86)\\Windows Kits\\8.1\\')]
mozbuild2.3:
[('KitsRoot10', u'C:\\Program Files\\Windows Kits\\10\\')]

This would be from different views of the registry being shown to 64-bit python, as opposed to the 32-bit python we're usually using.

C:\Program Files\Windows Kits\10\ doesn't really contain anything, certainly no include or library folder.
My guess is that since visual studio is 32-bit, all the useful stuff is stashed in 'Program Files (x86)', not the 64-bit 'Program Files'.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Doesn't the build system already properly detect the Win10 SDK? Anyway, in general my response to these kinds of issues "File a build system bug" as that's the right place to do this work now. We explicitly want to move away from MozillaBuild being the source of truth for detecting the environment around it.

Comment 7

[Kelsey Gilbert [:jgilbert]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #6)
> Doesn't the build system already properly detect the Win10 SDK? Anyway, in
> general my response to these kinds of issues "File a build system bug" as
> that's the right place to do this work now. We explicitly want to move away
> from MozillaBuild being the source of truth for detecting the environment
> around it.

Sure. I have a patch. I'll file a proper bug for this and post a patch tomorrow.

Comment 8

[(not currently active) Ted Mielczarek]

Is there anything you need from me here? Sounds like jgilbert already has a patch for that issue.

This would also have fixed the issue I hit in bug 1344790, where the zstd-compressed bundles we serve from hg.mo could not be decompressed with a 32-bit Python.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

No, I think we're good here given Jeff's bug having been filed. Per the dev-builds email I sent a month or so ago, I am going to move forward with making MozillaBuild 64-bit only.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

(In reply to Gregory Szorc [:gps] from comment #15)
> >  python_installer = "python-2.7.13.msi"
> > +check_call(["msiexec.exe", "/q", "/a", join(sourcedir, python_installer),
> > +            "TARGETDIR=" + python27_dir])
> 
> Eh? I know for a fact the Python installer can run in a mode that doesn't
> pollute the system. Is this a matter of passing a new flag to the installer?

Greg, which options did you have in mind? I mainly want to avoid adding any superfluous registry entries, add/remove programs entries, registering file extensions, etc. AFAICT, the administrative installation option still fits the bill best.
https://www.python.org/download/releases/2.4/msi/

Comment 11

[Gregory Szorc [:gps]]

I agree the "administrative installation" seems to fit our use case. We don't want registry settings, etc from things that MozillaBuild installs: MozillaBuild should be as self-contained and non-invasive as possible.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/369fbbda6317
Use the 64-bit build of python 2.7.13.
https://hg.mozilla.org/mozilla-build/rev/ac5c51ccf93d
Update the bundled sqlite3 DLL to 3.19.2 64-bit.