870420 - Require Python 2.7.3 to build the tree

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

We keep running into subtle bugs fixed in Python 2.7.3 and most of the world has been running 2.7.3 for seemingly ages. I'd like to bump the build system requirements to 2.7.3. It's quite likely the build will continue to work on 2.7.2 and lower. But, I have little interest in continuing to support it.

MozillaBuild was previously holding us back. And, now 1.7 is out with Python 2.7.4. So, I propose we give people a few days to upgrade then we move forward.

Any objections?

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

You should respond to the .platform thread saying that this version of mozillabuild will be required soon.  Also check what the infrastructure (including the thunderbird stuff) is running.

That said, full speed ahead.

Comment 2

[Gregory Szorc [:gps]]

Looks like the build infra still has some stragglers on 2.7.2 :(

https://tbpl.mozilla.org/?tree=Try&rev=b53497c4d8eb

Comment 3

[Chris Peterson [:cpeterson]]

btw, Python 2.7.2 is the default on OSX 10.8.3, but Python 2.7.4 is easily installable from Homebrew.

Comment 4

[Gregory Szorc [:gps]]

(In reply to Chris Peterson (:cpeterson) from comment #3)
> btw, Python 2.7.2 is the default on OSX 10.8.3, but Python 2.7.4 is easily
> installable from Homebrew.

This is why we have in-tree system bootstrapping code. One line command and your system is configured to build the tree. |mach bootstrap| is the easy way to run it.

Comment 5

[Gregory Szorc [:gps]]

Attachment: Require Python 2.7.3 to build, v1

I figure I'll get review now to avoid lag later. Will hold off deploying until automation is upgraded or people have a week or two to upgrade MozillaBuild, whichever comes first.

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 747555 [details] [diff] [review]
Require Python 2.7.3 to build, v1

Review of attachment 747555 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/virtualenv/populate_virtualenv.py
@@ +367,5 @@
> +        log_handle.write('Python %s or greater (but not Python 3) is '
> +            'required to build. ' % MINIMUM_PYTHON_VERSION)
> +        log_handle.write('You are running Python %s.\n' % our)
> +
> +        if os.name in ('nt', 'ce'):

I'm presuming the 'ce' case is for WindowsCE? If so, we don't support it any more (bug 614720 and dependants).

Comment 7

[(not currently active) Ted Mielczarek]

Comment on attachment 747555 [details] [diff] [review]
Require Python 2.7.3 to build, v1

Review of attachment 747555 [details] [diff] [review]:
-----------------------------------------------------------------

Fine with me, once you get the build slaves sorted out.

Do our bootstrap scripts ensure that they're installing Python > 2.7.3? If not we should make sure that's fixed before landing this as well.

Comment 8

[Axel Hecht [:Pike]]

Should we do this while bug 871817 waits for a fix? I.e., 2.7.4 showing bugs

Comment 9

[Gregory Szorc [:gps]]

(In reply to Axel Hecht [:Pike] from comment #8)
> Should we do this while bug 871817 waits for a fix? I.e., 2.7.4 showing bugs

Maybe. What's the impact of bug 871817? If it affects most normal developers, then we should probably block on it. If only a small number of people actually build the Firefox OS simulator locally, then I'm not too worried.

Comment 10

[Francesco Lodolo [:flod]]

Issue is "Python 2.7.4 breaks ZipFile extraction of zip files with unicode member paths", not sure if that's limited only to Addon-sdk.

Comment 11

[Gregory Szorc [:gps]]

MozillaBuild has Python 2.7.5 now, so I think we're unblocked.

https://tbpl.mozilla.org/?tree=Try&rev=48fa9e76cdc2

Comment 12

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/df244fde6aa4

Finally. (I pushed before try was done, but builds had been running for a while, so they presumably passed configure.)

Comment 13

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/df244fde6aa4

Comment 14

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/a2c84945e1ef

Comment 15

[Phil Ringnalda (:philor)]

That being the merge of a backout that seems to have forgotten to reopen.

Comment 16

[Ed Morley [:emorley]]

(In reply to Phil Ringnalda (:philor) from comment #15)
> That being the merge of a backout that seems to have forgotten to reopen.

EKR, can you explain the reason for the backout? (There was no comment in the bug)

Cheers :-)

Comment 17

[Eric Rescorla (:ekr)]

Ed,

A lot of people don't have Python 2.7.3 (for instance, MacOS Mountain
Lion comes with 2.7.2, DougT reports that mach bootstrap does not
upgrade as advertised, and gps was not around.

See http://logbot.glob.com.au/?c=mozilla%23developers&s=7+Sep+2013&e=7+Sep+2013


14:52	dougt	gps: yeah, i am broken too.
14:52	dougt	./mach bootstrap doesn't work
14:52	dougt	and I didn't remember seeing a .platform post about this change (although, i could have missed it)
14:52	dougt	i suspect that there are going to be a bunch of people broken by this change (and given that there is a B2G work week starting basically now), I'd like instructions on how to fix my build env, or a backout.
14:54	Rik	dougt: gps sent something on the 5th
14:55	dougt	yup, i see that now. :)
14:55	ekr	dougt: looking at the patch, it appears that the only change is to *enforce 2.7.3, so I think if you just do a local backout you will be left in a temporarily working state, until someone actually uses a 2.7.3 feature
14:55	ekr	(I agree that this is unsatisfactory going forward)
14:56	dougt	patches accepted.
14:56	ekr	I'm trying now. Will post the patch if it works
15:10	bsmedberg	dougt: gps is away for the weekend, please back him out
15:11	bsmedberg	or at least, please feel free to back him out ;-)


IMO when this patch is re-landed it should come with a facility to let you specify
a specific python path rather than requiring 2.7.3 to be first in your path.
If such a facility exists, it should be documented and called out in the error
message.

Comment 18

[Ed Morley [:emorley]]

Thank you for the explanation.

(I wasn't disputing the reasoning, more the fact no comment had been left in the bug...)

Comment 19

[Ed Morley [:emorley]]

Sorry meant to add - are there bugs filed for what isn't working with mach bootstrap? :-)

Comment 20

[Eric Rescorla (:ekr)]

(In reply to Ed Morley [:edmorley UTC+1] from comment #19)
> Sorry meant to add - are there bugs filed for what isn't working with mach
> bootstrap? :-)

This was reported by DougT. Added him with needinfo

Comment 21

[Justin Wood (:Callek)]

I'll note here I was simultaneously asking to backout from :ted while gps was away due to SeaMonkey bustage.

Ted wanted a "finite date we can reland" though to agree to it, which I was suggesting at the time "Gecko 27" (as in, at least 1 day after gecko 24's release).

I'm happy to have more time if the mach bootstrap issue is another driver for delaying this, but wanted to cite my request/reason in bug incase the mach issue is fixed sooner

Comment 22

[Gregory Szorc [:gps]]

mach bootstrap absolutely works for OS X/homebrew as of bug 872231.

SeaMonkey has had *months* to prep their automation for this.

There are a few things we can do to make bootstrap and the user experience better (e.g. checking for /usr/local/bin on $PATH before /usr/bin). But I think we've put in enough effort and have waited enough to land this. What more needs to be done? At some point we need to bite the bullet and land this.

Comment 23

[Benjamin Smedberg]

Considering that this change broke dougt's build and mach bootstrap didn't work, perhaps we should understand what's going on with it. This change doesn't appear to be urgent or blocking things.

Comment 24

[Gregory Szorc [:gps]]

We knew people would need to take action to upgrade Python. We sent out an email to dev-platform about this intended change months ago. I thought people had enough time to take action, that's all.

It is true this isn't technically blocking anything. But we still periodically run into Python bugs fixed in 2.7.3 and it's annoying to have to apply the same fixes and workarounds everywhere.

FWIW, this patch impacts more than just the build - it impacts people in ATeam. They write much more Python than the build system and are impacted by this much more. They would love 2.7.3+ everywhere, including on the test slaves. This patch represents the beginning of a happier world for everyone who writes Python in support of Firefox.

Let's plan to land this again when 27 is cut. I'll also file some bugs to make the bootstrap experience better. Hopefully those will land by 27. If not, we can prominently post links to http://lmgtfy.com/?q=install+python+2.7.5+on+os+x

Comment 25

[Doug Turner (:dougt)]

`mach bootstrap` fails for me.  I don't have time to debug.  We should break developers for non-critical version updates.

http://pastebin.mozilla.org/3011097

Comment 26

[Doug Turner (:dougt)]

should not ^

Comment 27

[Eric Rescorla (:ekr)]

(In reply to Gregory Szorc [:gps] from comment #24)
> We knew people would need to take action to upgrade Python. We sent out an
> email to dev-platform about this intended change months ago. I thought
> people had enough time to take action, that's all.
> 
> It is true this isn't technically blocking anything. But we still
> periodically run into Python bugs fixed in 2.7.3 and it's annoying to have
> to apply the same fixes and workarounds everywhere.
> 
> FWIW, this patch impacts more than just the build - it impacts people in
> ATeam. They write much more Python than the build system and are impacted by
> this much more. They would love 2.7.3+ everywhere, including on the test
> slaves. This patch represents the beginning of a happier world for everyone
> who writes Python in support of Firefox.
> 
> Let's plan to land this again when 27 is cut. I'll also file some bugs to
> make the bootstrap experience better. Hopefully those will land by 27. If
> not, we can prominently post links to
> http://lmgtfy.com/?q=install+python+2.7.5+on+os+x

From my perspective, it's not really OK to force people to have a new
python anywhere on their path (and especially not first) in order 
to build Firefox. I would suggest allowing a MOZ_PYTHON env variable
or the like.

The problem isn't that I don't know how to build python 2.7.5. It's that
I don't want to run it by default.

Comment 28

[Nathan Froyd [:froydnj]]

(In reply to Eric Rescorla (:ekr) from comment #27)
> From my perspective, it's not really OK to force people to have a new
> python anywhere on their path (and especially not first) in order 
> to build Firefox. I would suggest allowing a MOZ_PYTHON env variable
> or the like.
> 
> The problem isn't that I don't know how to build python 2.7.5. It's that
> I don't want to run it by default.

Just set PYTHON in your mozconfig or provide it as an environment variable when you run configure.

Comment 29

[Gregory Szorc [:gps]]

What Nathan said.

Or we can have the bootstrapper install Python from source into a well-defined location (as proposed in bug 914372) and have configure look there first. I think that would facilitate "it just works" and would make a lot of people happy.

This is another reason why I want Mozilla to publish archives of the chroot environment used to produce official builds (bug 896023). If we can give people an isolated environment to perform builds, I don't think people will complain too much about "system pollution" from installed dependencies, etc. While it has its downsides, you can't argue it delivers a nearly fully isolated environment, mitigating almost all "don't touch my system config, bro" complaints.

Comment 30

[Doug Turner (:dougt)]

that really isn't a great solution.  the point is to make building easier.  I do not see how adding a new variable to mozconfig helps with this effort.

exactly why are we upgrading?  Per comment #24, maybe this should be WONTFIX?

Comment 31

[Gregory Szorc [:gps]]

We're upgrading to make the cost of developing Python tools for the build and automation infrastructure easier. We're trading a one-time inconvenience for some developers for less headaches making build and automation tools better.

The longer we wait with this transition, the more and more people will be unable to write Python that is dual Python 2/3 compatible, which will only add to the cost for conversion later. This is due to bugs in Python 2.7.2 and below that make dual compatibility extremely difficult.

Comment 32

[Doug Turner (:dougt)]

Where are the headaches?  Bug numbers, please.

Look, i don't really care if you upgrade or not -- but we need to make damn sure that the developers building on the Mac don't *inconvenienced*.

Comment 33

[Gregory Szorc [:gps]]

Mac users are already inconvenienced in that the Xcode-provided Clang doesn't build the tree for OS X up until 10.7 IIRC!

Two years ago we didn't have an in-tree bootstrap tool and people were left on their own, following often out-of-date instructions on a wiki to install system dependencies.

On my first day at Mozilla my fresh MBP couldn't build m-c because the tree wasn't yet compatible with OS X 10.7 because of a bug in grep (bug 673209).

There has never been an expectation of a turnkey build experience (despite my efforts). I think we should focus our energy on making the tools better.

We need to require Python 2.7.3 to stop accumulating technical debt for working around bugs in 2.7.2 and for future compatibility with Python 3. We will deploy better tools to make this transition less painful and we will reland this patch.

Comment 34

[Eric Rescorla (:ekr)]

(In reply to Nathan Froyd (:froydnj) from comment #28)
> (In reply to Eric Rescorla (:ekr) from comment #27)
> > From my perspective, it's not really OK to force people to have a new
> > python anywhere on their path (and especially not first) in order 
> > to build Firefox. I would suggest allowing a MOZ_PYTHON env variable
> > or the like.
> > 
> > The problem isn't that I don't know how to build python 2.7.5. It's that
> > I don't want to run it by default.
> 
> Just set PYTHON in your mozconfig or provide it as an environment variable
> when you run configure.


I see. Well, it would have been awfully nice if the error message that the
build had spit that out instead of just telling me to upgrade.

Comment 35

[Gregory Szorc [:gps]]

It was *supposed* to spit out a helpful error message. You obviously hit a bug, probably due to a not-yet-encountered corner case. It's extremely difficult to catch all these bugs until things are deployed in the field because all systems are slightly different.

Comment 36

[Nathan Froyd [:froydnj]]

(In reply to Gregory Szorc [:gps] from comment #35)
> It was *supposed* to spit out a helpful error message. You obviously hit a
> bug, probably due to a not-yet-encountered corner case. It's extremely
> difficult to catch all these bugs until things are deployed in the field
> because all systems are slightly different.

Presumably you mean this:

http://mxr.mozilla.org/mozilla-central/source/build/autoconf/python-virtualenv.m4#14

?

Comment 37

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/282144602728

Comment 38

[Doug Turner (:dougt)]

gps, what is the upgrade path on the mac?

Comment 39

[Gregory Szorc [:gps]]

(In reply to Doug Turner (:dougt) from comment #38)
> gps, what is the upgrade path on the mac?

$ mach bootstrap

(The build system output will tell you this when it fails.)

Comment 40

[Doug Turner (:dougt)]

pastebin.mozilla.org/3090051

This is broken.  I suspect other developers building on the mac will hit the same problem.

Comment 41

[Gregory Szorc [:gps]]

(In reply to Doug Turner (:dougt) from comment #40)
> pastebin.mozilla.org/3090051
> 
> This is broken.  I suspect other developers building on the mac will hit the
> same problem.

We should have fixed that on current inbound. Please confirm you are running 282144602728 or newer.

Comment 42

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/282144602728

Comment 43

[Doug Turner (:dougt)]

~/builds/mozilla-central/ $ ./mach bootstrap

Looks like you have Homebrew installed. We will install all required packages via Homebrew.


Your environment's PATH variable lists a system path directory (/usr/bin)
before the path to your package manager's binaries (/usr/local/bin).
This means that the package manager's binaries likely won't be
detected properly.

Please modify your shell's configuration (e.g. ~/.bash_profile) to
have /usr/local/bin appear in $PATH before /usr/bin. e.g.

    export PATH=/usr/local/bin:$PATH

~/builds/mozilla-central/ $     export PATH=/usr/local/bin:$PATH
~/builds/mozilla-central/ $ ./mach build
 0:00.16 /usr/bin/make -f client.mk -s
 0:00.45 Clobber not needed.
 0:00.47 usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
 0:00.47        cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
 0:00.48 cd obj-x86_64-apple-darwin12.4.0
 0:00.48 /Users/dougt/builds/mozilla-central/configure
 0:00.61 loading cache ./config.cache
 0:00.69 checking host system type... x86_64-apple-darwin12.4.0
 0:00.72 checking target system type... x86_64-apple-darwin12.4.0
 0:00.74 checking build system type... x86_64-apple-darwin12.4.0
 0:00.74 checking for mawk... (cached) gawk
 0:00.75 checking for python2.7... (cached) /usr/bin/python2.7
 0:00.75 Creating Python environment
 0:00.78 Python 2.7.3 or greater (but not Python 3) is required to build. You are running Python 2.7.2.
 0:00.78 Run |mach bootstrap| to ensure your system is up to date.
 0:00.78 
 0:00.78 If you still receive this error, your shell environment is likely detecting
 0:00.78 another Python version. Ensure a modern Python can be found in the paths
 0:00.78 defined by the $PATH environment variable and try again.
 0:00.79 ------ config.log ------
 0:00.79 This file contains any messages produced by compilers while
 0:00.79 running configure, to aid debugging if configure makes a mistake.
 0:00.79 
 0:00.79 configure:1110: checking host system type
 0:00.79 configure:1131: checking target system type
 0:00.79 configure:1149: checking build system type
 0:00.79 configure:1224: checking for mawk
 0:00.79 configure:1298: checking for python2.7
 0:00.79 *** Fix above errors and then restart with               "/usr/bin/make -f client.mk build"
 0:00.79 make[2]: *** [configure] Error 1
 0:00.79 make[1]: *** [obj-x86_64-apple-darwin12.4.0/config.status] Error 2
 0:00.79 make: *** [build] Error 2
 0:00.84 295 compiler warnings present.
~/builds/mozilla-central/ $ 


~/builds/mozilla-central/ $ ./mach bootstrap

Looks like you have Homebrew installed. We will install all required packages via Homebrew.

Your version of Mercurial (2.7) is sufficiently modern.
Homebrew 0.9.4
git checkout -q master 
==> git rev-parse -q --verify HEAD
a749caea9716c1f65e02f0716b9aea69c57ec0fa
git config core.autocrlf false 
git pull origin refs/heads/master:refs/remotes/origin/master
Already up-to-date.
==> git rev-parse -q --verify HEAD
a749caea9716c1f65e02f0716b9aea69c57ec0fa
Already up-to-date.
Homebrew 0.9.4
Error: python-2.7.5 already installed
Kernel.exit
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

CalledProcessError: Command '[u'/usr/local/bin/brew', u'-v', u'upgrade', u'python']' returned non-zero exit status 1

  File "/Users/dougt/builds/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 24, in bootstrap
    bootstrapper.bootstrap()
  File "/Users/dougt/builds/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 87, in bootstrap
    instance.ensure_python_modern()
  File "/Users/dougt/builds/mozilla-central/python/mozboot/mozboot/base.py", line 261, in ensure_python_modern
    self.upgrade_python(version)
  File "/Users/dougt/builds/mozilla-central/python/mozboot/mozboot/osx.py", line 389, in upgrade_python
    self._upgrade_package('python')
  File "/Users/dougt/builds/mozilla-central/python/mozboot/mozboot/osx.py", line 378, in _upgrade_package
    subprocess.check_call([self.brew, '-v', 'upgrade', package])
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)


This is busted for me.

Comment 44

[Gregory Szorc [:gps]]

File a new bug for bootstrap bustage.

I'm pretty sure the bug you just reported was fixed ~24 hours ago in inbound. Are you *sure* you are running a modern tree?

Comment 45

[Gian-Carlo Pascutto [:gcp]]

As someone on Ubuntu Natty (Python 2.7.1), this hit pretty hard. Even the common fkrull/deadsnakes PPA repos don't have any updates for Python sub-releases, there's no Python 2.8 and Python 3+ doesn't work for building Firefox. I couldn't find any other other usable PPAs either.

I have no idea how to roll my own PPA and didn't want to dive into that in the middle of a workweek, so I ended up using this:
https://github.com/yyuu/pyenv

Which allows you to use python 2.7.5 in your mozilla directories and the system python everywhere else.

Comment 46

[Doug Turner (:dougt)]

To fix the mac bustage, i think you need:
    brew link --overwrite python OR brew unlink python && brew link python

Comment 47

[Gregory Szorc [:gps]]

If you can come up with a way to detect when brew's python isn't linked properly, file a bug and we'll add it to bootstrap. Otherwise, I'm inclined to file this issue under "user error." AFAIK you were the only person to report a bootstrap issue on OS X.

I want to reiterate, the in-tree bootstrap script is best effort. It doesn't cover every corner case. Never will. 2 years ago we didn't have an in-tree bootstrap script. We rely on people like you to report corner cases so we can plug holes in the support.

Comment 48

[Mike Hommey [:glandium]]

(In reply to Doug Turner (:dougt) from comment #43)
>  0:00.75 checking for python2.7... (cached) /usr/bin/python2.7

The key to your failure is this "(cached)" thing. Blow up $objdir/config.cache. mach bootstrap should probably do that for you (in which case filing a bug for that would be the right thing to do.

Comment 49

[[e-mail]]

(In reply to Mike Hommey from comment #48)
> (In reply to Doug Turner from comment #43)
> >  0:00.75 checking for python2.7... (cached) /usr/bin/python2.7
> 
> The key to your failure is this "(cached)" thing. Blow up
> $objdir/config.cache. mach bootstrap should probably do that for you (in
> which case filing a bug for that would be the right thing to do.

If it upgrades your python it should probably also blow away your _virtualenv too.