758694 - Fix PATH on Lion build slaves (virtualenv install fails)

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[(not currently active) Ted Mielczarek]

Nightly builds have:
PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin

but repacks have:
PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin

The swapping of the first two entries means we pick up the python in /tools/buildbot/bin, which is already a virtualenv, which breaks the build.

Comment 1

[Mark Banner (:standard8)]

The Mac 32 bit builders on all branches also have

PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin

so they need fixing as well.

Comment 2

[Mark Banner (:standard8)]

and the repacks issue affects Firefox as well, just in case we're any different there.

Comment 3

[(not currently active) Ted Mielczarek]

Firefox repacks looked to be broken in a different way, but I may not have been looking at all the right logs.

Comment 4

[Mark Banner (:standard8)]

Ok, I've done some more investigation. According to the logs on Mozilla-l10n and MozillaTest, this specific issue just affects the Mac builders.

As it turns out, this failure doesn't about configure, so actually these builders will continue to run with the failure being unnoticed. I've filed bug 758723 on the configure issue.

Comment 5

[Mark Banner (:standard8)]

Oh and I believe this is both the 10.5 & 10.7 builders.

Looking at the path settings we already have some overrides, don't know if it is appropriate to copy those or not:

http://mxr.mozilla.org/build/search?string=PATH&case=1&find=%2Fmozilla%2F.*config.py&findi=&filter=^[^\0]*%24&hitlimit=&tree=build

Comment 6

[Aki Sasaki (not active)]

(In reply to Ted Mielczarek [:ted] from comment #0)
> Nightly builds have:
> PATH=/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:/bin:/usr/
> sbin:/sbin:/usr/local/bin:/usr/X11/bin

Where are you seeing this?
I only see /tools/buildbot/bin:... for PATH on all lion builds/repacks.

Comment 7

[(not currently active) Ted Mielczarek]

I don't remember offhand, I think Mark linked me to a nightly build log or something? This was probably a Thunderbird slave.

Comment 8

[(not currently active) Ted Mielczarek]

FWIW, having the virtualenv python first definitely breaks our build. (It's just not failing in a way that makes the build burn right now.)

Comment 9

[Mark Banner (:standard8)]

See the start of the build step here ("Started compile failed"):

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm-central-macosx-debug/1338823020/comm-central-macosx-debug-bm30-build1-build92.txt.gz

and in this log, look in the block just after "configure done":

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1338818456.1338819450.27160.gz&fulltext=1#err1

Comment 10

[Aki Sasaki (not active)]

(In reply to Mark Banner (:standard8) from comment #9)
> See the start of the build step here ("Started compile failed"):
> 
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm-
> central-macosx-debug/1338823020/comm-central-macosx-debug-bm30-build1-
> build92.txt.gz
> 
> and in this log, look in the block just after "configure done":
> 
> http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1338818456.
> 1338819450.27160.gz&fulltext=1#err1

Those both have /tools/buildbot/bin as the first entry in PATH.  I'm asking for an example where it's /tools/python/bin, since comment 0 says there's a mix and I'm not finding it.

Comment 11

[Mark Banner (:standard8)]

(In reply to Aki Sasaki [:aki] from comment #10)
> Those both have /tools/buildbot/bin as the first entry in PATH.  I'm asking
> for an example where it's /tools/python/bin, since comment 0 says there's a
> mix and I'm not finding it.

Oh sorry, in which case, I think comment 0 was referring to the Linux builders.

Comment 12

[Aki Sasaki (not active)]

Attachment: potential fix

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 630272 [details] [diff] [review]
potential fix

http://hg.mozilla.org/build/buildbot-configs/rev/6aaa306e8abc

Comment 14

[Armen [:armenzg]]

The changed got merged to the production branch and reconfiguration happened at 6:15 AM PDT.

Comment 15

[Mark Banner (:standard8)]

I think we missed the macosx-debug builds, this one still has a broken path:

https://tbpl.mozilla.org/php/getParsedLog.php?id=12418246&full=1&branch=comm-central

Comment 16

[Aki Sasaki (not active)]

Attachment: get macosx64, macosx64-debug, and macosx too

Comment 17

[Aki Sasaki (not active)]

Comment on attachment 630802 [details] [diff] [review]
get macosx64, macosx64-debug, and macosx too

http://hg.mozilla.org/build/buildbot-configs/rev/225c93558cea

Comment 18

[Armen [:armenzg]]

Just FYI the fix might fix the L10n issues that Mac builds are hitting:
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-l10n-hy-AM/1338999163.1338999222.4577.gz&fulltext=1#err0
IOError: [Errno 13] Permission denied: './_virtualenv/lib/python2.7/distutils/__init__.py'

Comment 19

[John Ford [:jhford] CET/CEST Berlin Time]

This wasn't done for B2G, and it started failing because of this and a landing on mozilla-central.

Comment 20

[Chris Cooper [:coop] (he/him)]

(In reply to John Ford [:jhford] from comment #19)
> This wasn't done for B2G, and it started failing because of this and a
> landing on mozilla-central.

Got a link to a failing build so I can make sure I update in the correct place?

Comment 21

[John Ford [:jhford] CET/CEST Berlin Time]

https://tbpl.mozilla.org/php/getParsedLog.php?id=16547765&tree=Firefox&full=1

The issue is that configure is finding "checking for python2.7... /tools/buildbot/bin/python2.7".  We don't want the /tools/buildbot python

Comment 22

[Donovan Preston [:fzzzy]]

This is blocking a bunch of webdevs that are onboarding to help with gaia. Webdevs that should be writing js are having to do hour long compiles and often having problems. Hint, hint, maybe it should be higher than p2? :-)

Comment 23

[John Ford [:jhford] CET/CEST Berlin Time]

part of the problem here is that these builds are considered unsupported in automation and as a result are always hidden on TBPL.  Unless we have per-checkin builds, it's very unlikely that these builds will be considered tier 1 or even unhidden.

It'd be great if we had something in between completely hidden and invisible and failure closing the tree, but we don't.  To get these builds to tier-1, we would need to have per-checkin builds happening and try support.  We'd get try support for free by having per-checkin builds.

Comment 24

[Chris Cooper [:coop] (he/him)]

(In reply to John Ford [:jhford] from comment #21)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=16547765&tree=Firefox&full=1
> 
> The issue is that configure is finding "checking for python2.7...
> /tools/buildbot/bin/python2.7".  We don't want the /tools/buildbot python

Finish the thought...which python version do you want/are expecting?

Comment 25

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Chris Cooper [:coop] from comment #24)
> (In reply to John Ford [:jhford] from comment #21)
> > https://tbpl.mozilla.org/php/getParsedLog.php?id=16547765&tree=Firefox&full=1
> > 
> > The issue is that configure is finding "checking for python2.7...
> > /tools/buildbot/bin/python2.7".  We don't want the /tools/buildbot python
> 
> Finish the thought...which python version do you want/are expecting?

Whichever one desktop uses, which appears to be "/tools/python/bin/python2.7"

Comment 26

[Chris Cooper [:coop] (he/him)]

(In reply to John Ford [:jhford] from comment #25) 
> Whichever one desktop uses, which appears to be "/tools/python/bin/python2.7"

Don't know whether it's intentional, but the b2g configs seem to be prefacing the PATH with /tools/buildbot/bin, e.g. 

https://hg.mozilla.org/build/buildbot-configs/file/91092188b466/mozilla/b2g_config.py#l82

I'm going to try appending /tools/buildbot/bin rather than prepending in staging to see whether that fixes the issue, or anything else ends up broken.

I would note that the base version of python is the same for /tools/buildbot and /tools/python (at least on Mac), so we're likely looking at a difference in installed modules. From the slave in the failing build above:

bld-lion-r5-058:~ cltbld$ which -a python
/tools/buildbot/bin/python
/tools/python/bin/python
/usr/bin/python
bld-lion-r5-058:~ cltbld$ /tools/buildbot/bin/python --version
Python 2.7.2
bld-lion-r5-058:~ cltbld$ /tools/python/bin/python --version
Python 2.7.2
bld-lion-r5-058:~ cltbld$ /usr/bin/python --version
Python 2.7.1

Comment 27

[Gregory Szorc [:gps]]

I think the "bad" Python is already in a virtualenv or has symlinks involved or something.

Comment 28

[John Ford [:jhford] CET/CEST Berlin Time]

/tools/buildbot is already a virtual env.  Moving /tools/buildbot to the end of PATH seems like a good idea, not sure why it'd be first.  Is it used for anything other than sendchanges?  The system and /tools/python/ interpreters shouldn't have anything in them that overrides the buildbot package in /tools/buildbot.

Comment 29

[Chris Cooper [:coop] (he/him)]

Attachment: Adjust PATH for b2g builds so /tools/python/bin is found before /tools/buildbot/bin

This generated green gecko builds for linux(32|64) and Mac in staging. 

Windows seems broken (sed syntax issue), but since this patch doesn't touch Windows, I'm assuming that's unrelated until someone tells me otherwise.

Comment 30

[Chris Cooper [:coop] (he/him)]

Comment on attachment 677241 [details] [diff] [review]
Adjust PATH for b2g builds so /tools/python/bin is found before /tools/buildbot/bin

https://hg.mozilla.org/build/buildbot-configs/rev/987c8a8a0e8b

Comment 31

[Chris Cooper [:coop] (he/him)]

I see this build from last night:

https://tbpl.mozilla.org/php/getParsedLog.php?id=16671104&tree=Firefox&full=1

Please re-open if more is required here.

Comment 32

[Donovan Preston [:fzzzy]]

Horray! Thank you!