611791 - Remove sys.path changes in mozharness

Status: RESOLVED WONTFIX

(Release Engineering :: Applications: MozharnessCore, defect, P5)

Description

[Aki Sasaki (not active)]

These are very handy in terms of not needing to futz with PYTHONPATH to get the libraries and configs in the search path.

However, this is messy and could potentially trip someone up who's expecting PYTHONPATH to work.

I'm hoping there's an elegant solution here I'm not thinking of.

Comment 1

[Mike Taylor [:bear]]

Are you loading these modules dynamically?  If so, then you can use the built-in __import__ function - but this also requires PYTHONPATH to be set or the modules to be in the same directory reachable as the main code:

http://docs.python.org/release/2.5.2/lib/built-in-funcs.html

i.e. if you main is foo/main.py and inside of foo is mods/bar/__init__.py then you would request __import__('mods.bar')

this is mostly from memory, so some tweaking may be required

Comment 2

[Aki Sasaki (not active)]

It's partially import, though that doc seems to say the libs need to be in subdirs of cwd or dirname(script).  Currently they're in dirname(script)/../lib/.

So changing the directory structure could be a fix.  __import__ could be a fix.
There's also the config file search path.

I'm trying to get setup+use of mozharness as low-touch as possible. Adding the sys.path tweaks allows one to just call the script and not worry about PYTHONPATH or setup.py or typing in long paths or cd'ing to specific directories or whatever.

So I'd like to find the best solution here, but am enjoying the effects altering sys.path gives me for the moment.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Without really knowing the context -- could adding setup.py help?

For hacking,
  python setup.py develop
works great - it manages to symlink things such that you can hack on the code in place, but it gets plugged into the search path.  Then in production,
  python setup.py install
gets it installed in the python environment for real.

This requires virtualenv, but virtualenv is *really* easy to use.

Comment 4

[Aki Sasaki (not active)]

I want to be able to pull any revision and run the script, over and over, 24/7.
Running setup.py that many times would make that more complex and potentially add cruft that breaks things or make it less predictable.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

That's what 'setup.py develop' lets you do.  I use it for buildbot all the time, switching branches, and everything.

Comment 6

[Aki Sasaki (not active)]

How would it work on a production buildbot slave?
Wouldn't the virtual env screw up the python environment for buildbot?
And how about scratchbox, which requires a different python environment?

I'm willing to let you take this bug and show me a working proof of concept.
I'm not convinced enough to go down this route myself.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Don't worry about it - I don't know the details of what you're working on, so I'm only a distraction here.  Sorry for the noise!

Comment 8

[Aki Sasaki (not active)]

I find the sys.path change removal requirement is amusing, now that I've seen Ben's scripts, which *also* alter sys.path, but were r+'ed by Catlee w/out comment :)

Comment 9

[Aki Sasaki (not active)]

I'm thinking that, instead of removing the sys.path lines immediately, put in a


try:
    import mozharness
except ImportError:
    #sys.path hackery

as a way to keep things working.

Comment 10

[Aki Sasaki (not active)]

From bug 700415 comment 69:

Is this a shortcut so that we don't have to have mozharness installed in the package index? If so you might want to use "os.path.dirname(os.path.realpath(__file__))" instead so it doesn't depend on where you run the script from.

Comment 11

[Chris Cooper [:coop] (he/him)]

Aki: is this still an issue today?

Comment 12

[Aki Sasaki (not active)]

This still exists today.  I don't think it's a big problem, but was asked to file it during a review.