721134 - configman needs to be installed from submodule

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

Just like we're installing isodate as a python dependency we need to install the configman that is linked as a submodule inside the socorro/configman/ directory.

Note that `pip install` won't suffice because it doesn't guarantee the exact version we have git submodule'd to. 

The command to run is::

 cd socorro/configman
 sudo python setup.py install

Comment 1

[Robert Helmer [:rhelmer]]

As a first pass, I suggest we do the following:

1) have a "configman" rule in the Makefile which:
1.1) inits/updates submodule
1.2) calls setup.py and installs configman to socorro root
2) have "install" rule depend on "configman" rule
3) modify "install" run to include configman in socorro root

This means that we'll get configman in /data/socorro/application/ (at the same level as the "socorro" library) and it should be on the PYTHONPATH without any further work, same for local installs.

Comment 2

[Robert Helmer [:rhelmer]]

Hmm so it's already installable from the submodule into a virtualenv, is what's really needed here a strategy for deploying to stage/prod?

I think the two options are:

a) in the socorro.tar.gz that gets unpacked in /data/socorro/ as in comment 1

or,

b) generate an RPM using e.g. pypi2rpm 

I think (a) might be a better choice for the moment, since I expect that configman will be moving quickly for a while and we'll need to keep things in sync.

Comment 3

[Robert Helmer [:rhelmer]]

I did not actually get this done today, will get it early in the next cycle so it can be used for 2.4.3

Comment 4

[Robert Helmer [:rhelmer]]

Going to do a bit of testing before I submit the pull request, but here is the change I have in mind:

https://github.com/rhelmer/socorro/compare/master...bug721134-install-configman

This simply installs configman from the submodule into $(PREFIX)/application/ so it's installed alongside the socorro lib, and will be on PYTHONPATH automatically. PREFIX is by default /data/socorro but can be overridden (this is how we build tarballs for deploying, "make install PREFIX=./socorro" then "tar zxcf socorro.tar.gz socorro/")

One thing I do not like about the current patch is that every time "make install" is run, "git submodule init" is updated also. I feel like the submodule should have init/update run only if it exists, and not mess with it otherwise, but not sure which is least surprising.

Comment 5

[Peter Bengtsson [:peterbe]]

(In reply to Robert Helmer [:rhelmer] from comment #4)
> Going to do a bit of testing before I submit the pull request, but here is
> the change I have in mind:
> 
> https://github.com/rhelmer/socorro/compare/master...bug721134-install-
> configman
> 
> This simply installs configman from the submodule into
> $(PREFIX)/application/ so it's installed alongside the socorro lib, and will
> be on PYTHONPATH automatically. PREFIX is by default /data/socorro but can
> be overridden (this is how we build tarballs for deploying, "make install
> PREFIX=./socorro" then "tar zxcf socorro.tar.gz socorro/")
> 
> One thing I do not like about the current patch is that every time "make
> install" is run, "git submodule init" is updated also. I feel like the
> submodule should have init/update run only if it exists, and not mess with
> it otherwise, but not sure which is least surprising.

Right. I don't think we should run the git submodule. Installation is usually not part of downloading the program you're about to install. I think the assumption should be that all submodules are up to date. 

Now, where is socorro itself cloned or pull? If jenkins pulls it, can we add ``--recursive`` to it so that we get the whole thing by default. 

I chalk this up to not understanding the context sufficiently. For example, in the Django project, running `git submodule update` is part of the build.sh script
https://github.com/mozilla/sheriffs/blob/develop/scripts/build.sh#L15

I guess what I object to is the overly explicit treatment of configman. What if one day we have 10-20 different submodules. Like the playdoh projects have.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to Peter Bengtsson [:peterbe] from comment #5)
> > One thing I do not like about the current patch is that every time "make
> > install" is run, "git submodule init" is updated also. I feel like the
> > submodule should have init/update run only if it exists, and not mess with
> > it otherwise, but not sure which is least surprising.
> 
> Right. I don't think we should run the git submodule. Installation is
> usually not part of downloading the program you're about to install. I think
> the assumption should be that all submodules are up to date. 
> 
> Now, where is socorro itself cloned or pull? If jenkins pulls it, can we add
> ``--recursive`` to it so that we get the whole thing by default. 


Oh I didn't realize we could do it that way; yes it's jenkins, I'll see if the git plugin supports that (can work around if not).


> I chalk this up to not understanding the context sufficiently. For example,
> in the Django project, running `git submodule update` is part of the
> build.sh script
> https://github.com/mozilla/sheriffs/blob/develop/scripts/build.sh#L15


We should just do it this way, I'll add a build.sh for jenkins to call. 


> I guess what I object to is the overly explicit treatment of configman. What
> if one day we have 10-20 different submodules. Like the playdoh projects
> have.


This is a good point, I can make this more generic as part of this bug. How do the playdoh projects submodules (apologies if you already showed me and I forgot :) )?

The olde-world socorro has a "thirdparty" directory where deps are directly checked in, however I'd like to make a break from this for configman, so we have to make it up as we go a bit. If we can do what other Mozilla projects do that's all the better.

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #6)
> (In reply to Peter Bengtsson [:peterbe] from comment #5)
> > > One thing I do not like about the current patch is that every time "make
> > > install" is run, "git submodule init" is updated also. I feel like the
> > > submodule should have init/update run only if it exists, and not mess with
> > > it otherwise, but not sure which is least surprising.
> > 
> > Right. I don't think we should run the git submodule. Installation is
> > usually not part of downloading the program you're about to install. I think
> > the assumption should be that all submodules are up to date. 
> > 
> > Now, where is socorro itself cloned or pull? If jenkins pulls it, can we add
> > ``--recursive`` to it so that we get the whole thing by default. 
> 
> 
> Oh I didn't realize we could do it that way; yes it's jenkins, I'll see if
> the git plugin supports that (can work around if not).

Hmm so "git clone --recursive" seems to check out all submodules on "no branch" (and the jenkins git plugin doesn't seem to understand it)

"git submodule update --init --recursive" after clone seems to be useful, we can have a build.sh which calls that.

Comment 8

[Peter Bengtsson [:peterbe]]

We can't use THIRDPARTY for configman since it's not a python module. It's just a sub-directory where the actual python module is located. 

As far as I know, `git clone --recursive ...` is the exact same as `git clone ...; cd repo; git submodule update --init --recursive`.
There might be a difference in what is staged/HEAD but at least the same files should be equal.

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to Peter Bengtsson [:peterbe] from comment #8)
> We can't use THIRDPARTY for configman since it's not a python module. It's
> just a sub-directory where the actual python module is located. 

Right, sorry for the confusion - I am trying to say I don't want to use "thirdparty" anymore, we should explicit deprecate it (ftr, what goes in there are just the installed libs, not a source dir like configman submodule).

> As far as I know, `git clone --recursive ...` is the exact same as `git
> clone ...; cd repo; git submodule update --init --recursive`.
> There might be a difference in what is staged/HEAD but at least the same
> files should be equal.

Huh you are right; I had read that the latter ended up with the submodules on master but that's obviously mistaken.

In any case, we went over this in irc and are going to move all the steps that jenkins currently does into a build.sh so it's versioned and discoverable and part of the socorro release process, then we can improve this iteratively as needed.

Comment 10

[Robert Helmer [:rhelmer]]

lonnen, peterbe - r? https://github.com/mozilla/socorro/pull/346

I did some cleanup in a separate revision first, and also split out the work jenkins does to a build.sh, that info should really be in a more discoverable place since it's useful to other socorro users.

Comment 11

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/ed057f098c8e24f588e7894ceda0499728dcada3
bug 721134 - cleanup

https://github.com/mozilla/socorro/commit/5d0ce2339782f53d105251926d6530e8cf556a6c
bug 721134 - install configman from submodule into /application, alongside socorro lib; move jenkins work into a build.sh, treat submodules generically

https://github.com/mozilla/socorro/commit/990f5c1425a8e96900fc6ce9a80dee219e0361f5
Merge pull request #346 from rhelmer/bug721134-install-configman

Bug721134 install configman