Last Comment Bug 758823 - configure creates egg-info directories in the source tree
: configure creates egg-info directories in the source tree
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: All All
: -- normal (vote)
: mozilla18
Assigned To: Gregory Szorc [:gps]
:
Mentors:
: 759303 (view as bug list)
Depends on:
Blocks: 661908 760094
  Show dependency treegraph
 
Reported: 2012-05-25 15:47 PDT by Mike Hommey [:glandium]
Modified: 2012-09-04 18:49 PDT (History)
10 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Don't create egg-info directories, v1 (2.25 KB, patch)
2012-08-29 10:47 PDT, Gregory Szorc [:gps]
ted: review+
Details | Diff | Review

Description Mike Hommey [:glandium] 2012-05-25 15:47:54 PDT
This is what is being created during configure. They are create in the source tree :(

other-licenses/simplejson-2.1.1/simplejson.egg-info/
testing/mozbase/manifestdestiny/ManifestDestiny.egg-info/
testing/mozbase/mozinfo/mozinfo.egg-info/
testing/mozbase/mozinstall/mozInstall.egg-info/
testing/mozbase/mozlog/mozlog.egg-info/
testing/mozbase/mozprocess/mozprocess.egg-info/
testing/mozbase/mozprofile/mozprofile.egg-info/
testing/mozbase/mozrunner/mozrunner.egg-info/
Comment 1 Ted Mielczarek [:ted.mielczarek] 2012-05-25 18:34:20 PDT
Carl: is there a way to prevent this? AFAIK they don't do anything useful.
Comment 2 Carl Meyer (:carljm) 2012-05-26 18:20:34 PDT
The egg-info dirs are created by setuptools when a package is installed in-place via "python setup.py develop" (aka "pip install -e"), to allow setuptools/pip to track the metadata about what package and version is installed. It's ok to just remove them after the fact; the only noticeable impact will be that the package will no longer show up in e.g. a "pip freeze" and pip will no longer be able to "pip uninstall" it (although it'll still be importable).

There's no way I know of to prevent them being created in the first place, other than to not do the develop/editable install and instead just do "python setup.py install" (or "pip install" without the "-e"). Unless these packages are actually being modified in-place during the build process, I don't think there's any need to be installing them as develop/editable.
Comment 3 Mark Capella [:capella] 2012-05-28 06:29:03 PDT
So I can wipe these from my local machine but they'll keep returning?
Comment 4 Ted Mielczarek [:ted.mielczarek] 2012-05-29 05:54:14 PDT
Yes.
Comment 5 Ted Mielczarek [:ted.mielczarek] 2012-05-29 05:56:24 PDT
Carl: I intentionally used "develop" because I do think it will be useful. While we have some third-party modules in the tree (like simplejson), we also have quite a few Mozilla-specific modules, and it seems pretty likely that some developers will wind up in an edit and test cycle. It'd be a lot nicer if they didn't have to refresh the virtualenv every time.
Comment 6 Ted Mielczarek [:ted.mielczarek] 2012-05-29 06:06:16 PDT
*** Bug 759303 has been marked as a duplicate of this bug. ***
Comment 7 Jeff Hammel 2012-05-29 08:56:17 PDT
Maybe worth adding to .hgignore?
Comment 8 Justin Lebar (not reading bugmail) 2012-05-29 09:01:36 PDT
(In reply to Jeff Hammel [:jhammel] from comment #7)
> Maybe worth adding to .hgignore?

As a temporary fix, sure.  But I think in general it's unfortunate to clutter the srcdir with products of the build.
Comment 9 Mike Hommey [:glandium] 2012-05-29 09:01:49 PDT
We also have mozilla specific modules without setup.py. In fact, I suspect most are this way. Are we going to need a setup.py for any such module? Can't we just install them with symbolic links?
Comment 10 Ted Mielczarek [:ted.mielczarek] 2012-05-29 09:41:28 PDT
The non-setup.py ones will be handled separately, yeah. I need to file a bug on that, but our options are: a) just symlink them into the virtualenv's site-packages dir, or b) stick a .pth file in the site-packages dir containing a path to the directory containing them, which gets it added to sys.path (leaning towards b), since we can install whole dirs of modules that way).
Comment 11 Jeff Hammel 2012-05-29 09:44:41 PDT
FWIW, the creation of in-situ .egg-info (and similar) files is parallel to the outputting of .pyc files in python, as noted in other bugs.  Ideally, I'd like to see neither of them in the source tree, but also AFAIK these aren't easily preventable (at least without drawbacks)
Comment 12 Justin Lebar (not reading bugmail) 2012-05-29 09:48:24 PDT
Well, the pyc files caused havoc a month or so ago; there's certainly a trade-off with keeping those in the source tree.  I hope the downside here isn't quite as bad!
Comment 13 Ian Bicking (:ianb) 2012-05-30 09:45:01 PDT
FWIW, I often check egg-info directories into source control even when I don't check in .pyc files, if I am checking in "installed" versions of libraries (i.e., in a vendor library situation).  But if I'm handling source packages I leave them be but add *.egg-info to hgignore.
Comment 14 Mark Hammond [:markh] 2012-05-30 19:46:26 PDT
FWIW, as a workaround, add '*.egg-info' to your .hgignore
Comment 15 Ted Mielczarek [:ted.mielczarek] 2012-06-07 09:19:59 PDT
We've .hgignored them, seems to be fine for now.
Comment 16 Gregory Szorc [:gps] 2012-08-29 10:43:26 PDT
Patch forthcoming that nukes these. I figured it was easier to reopen this bug then file yet another one.
Comment 17 Gregory Szorc [:gps] 2012-08-29 10:47:20 PDT
Created attachment 656515 [details] [diff] [review]
Don't create egg-info directories, v1

Pythonistas may scream at this. But, it is exactly what we want:

1) Changes to srcdir don't require virtualenv repopulation (like |python setup.py develop|)
2) No extra crap in the srcdir

I think the only contentious part is removing the entries from the .hgignore and .gitignore files. I think this is a good idea because I maintain that the smaller these files are, the better: we shouldn't be spewing junk in the srcdir and I welcome any chance to purge my tree of things that aren't needed.

Try at http://tbpl.mozilla.org/?tree=Try&rev=0d941ca66e96
Comment 18 Gregory Szorc [:gps] 2012-08-29 11:20:56 PDT
If compiling C extensions, I would probably need to add --inplace to |python setup.py build_ext|. However, the only package we currently have with a C extension is simplejson. And, it looks like its setup.py file has been hacked up to never compile the C extension. So, I think I'll just remove the first line from packages.txt. I won't bother with a new patch.

Also, :sid0 says to ignore the Win64 try failure (pymake issue, I guess).
Comment 19 Ted Mielczarek [:ted.mielczarek] 2012-09-04 06:36:45 PDT
Comment on attachment 656515 [details] [diff] [review]
Don't create egg-info directories, v1

Review of attachment 656515 [details] [diff] [review]:
-----------------------------------------------------------------

Seems fine.
Comment 20 Gregory Szorc [:gps] 2012-09-04 10:47:02 PDT
https://hg.mozilla.org/integration/mozilla-inbound/rev/d034d1924fc7

I added instructions on how to remove egg-info directories to the commit message.
Comment 21 Ryan VanderMeulen [:RyanVM] 2012-09-04 18:49:50 PDT
https://hg.mozilla.org/mozilla-central/rev/d034d1924fc7

Note You need to log in before you can comment on or make changes to this bug.