Closed Bug 1161812 Opened 10 years ago Closed 10 years ago

server_status cannot find *_revision.txt files when running from package

Categories

(Socorro Graveyard :: Middleware, defect)

Product:
Socorro Graveyard
Component:
Middleware
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rhelmer, Assigned: rhelmer)

References

Details

Now that we install a Socorro package and run things out of that, the way we're getting socorro_revision.txt and breakpad_revision.txt into server_status doesn't work, since it is looking in os.path.dirname(__file__) We need to either put these .txt files into the package, or put them in some well-known location in the filesystem.
So these .txt files are not included anywhere at the moment? Meaning, it's not a matter of changing how the webapp opens it.
(In reply to Peter Bengtsson [:peterbe] from comment #1) > So these .txt files are not included anywhere at the moment? Meaning, it's > not a matter of changing how the webapp opens it. They are included, the problem is that the code is now running from a package deep inside the virtualenv so os.path.dirname(__file__) won't find it. The files go to /data/socorro/application/socorro/external/postgresql/
The webapp could potentially do something like: >>> import socorro >>> os.path.join(socorro.__path__[0], 'external/postgresql') Would that work? I'm sure we can hack anything we like. Even put it into settings but it would be nice if you don't get an OSError when you try to run this locally.
(In reply to Peter Bengtsson [:peterbe] from comment #3) > The webapp could potentially do something like: > > >>> import socorro > >>> os.path.join(socorro.__path__[0], 'external/postgresql') > > Would that work? > > I'm sure we can hack anything we like. Even put it into settings but it > would be nice if you don't get an OSError when you try to run this locally. (As we discussed in IRC) I don't think it will, because it'll return some weird path deep inside the virtualenv, here's what it does on my laptop: >>> import socorro,os >>> os.path.join(socorro.__path__[0], 'external/postgresql') '/Users/rhelmer/.virtualenvs/socorro/lib/python2.7/site-packages/socorro-master-py2.7.egg/socorro/external/postgresql' I'd expect it to be the same on prod but in /data/socorro/socorro-virtualenv instead of my ~/.virtualenv Probably would be best to include these .txt fils in the package at some location we can find, https://docs.python.org/2/distutils/sourcedist.html#the-manifest-in-template possibly? It looks like to access resources in the package we can use: >>> from pkg_resources import resource_string >>> socorro_revision = resource_string('socorro', 'socorro_revision.txt') I did a quick local test and this seems to work.
The point I tried to make was that if you include the *.txt files in the MANIFEST.in then they'd be put in `/Users/rhelmer/.virtualenvs/socorro/lib/python2.7/site-packages/socorro-master-py2.7.egg/socorro` and that's fine. I think I don't understand where and at what point the .txt files are created/edited. Are they created at the point of where we run `python setup.py build` or something?
(In reply to Peter Bengtsson [:peterbe] from comment #5) > The point I tried to make was that if you include the *.txt files in the > MANIFEST.in then they'd be put in > `/Users/rhelmer/.virtualenvs/socorro/lib/python2.7/site-packages/socorro- > master-py2.7.egg/socorro` and that's fine. > > I think I don't understand where and at what point the .txt files are > created/edited. Are they created at the point of where we run `python > setup.py build` or something? Right now they get created in scripts/install.sh but I was thinking it'd be better if the setup.py created them.
(In reply to Robert Helmer [:rhelmer] from comment #6) > (In reply to Peter Bengtsson [:peterbe] from comment #5) > > The point I tried to make was that if you include the *.txt files in the > > MANIFEST.in then they'd be put in > > `/Users/rhelmer/.virtualenvs/socorro/lib/python2.7/site-packages/socorro- > > master-py2.7.egg/socorro` and that's fine. > > > > I think I don't understand where and at what point the .txt files are > > created/edited. Are they created at the point of where we run `python > > setup.py build` or something? > > Right now they get created in scripts/install.sh but I was thinking it'd be > better if the setup.py created them. Well let me amend that - socorro_revision.txt is created by scripts/install.sh and is just basically: git rev-parse HEAD > $BUILD_DIR/socorro_revision.txt The breakpad revision is created by https://raw.githubusercontent.com/mozilla/socorro/master/scripts/build-breakpad.sh (which is what "make breakpad" runs)
We *also* get a JENKINS_BUILD, it'd be really great to get this onto the /status page (and in the status JSON) while we're at it!
So after these install.sh and build-breakpad.sh you later run the `python setup.py build`? If so, I'd suggest adding `include /socorro_revision.txt` to the MANIFEST.in file inside the socorro package so that when packaged, it becomes part of the build. For example, configman extracts its version number from a `version.txt` file that sits inside the package root (not the project root!) and it gets included in the tarball predictably thanks to this line https://github.com/mozilla/configman/blob/7df78c569ec93d37443297b256d7e14fb7a6a582/MANIFEST.in#L4
(In reply to Peter Bengtsson [:peterbe] from comment #9) > So after these install.sh and build-breakpad.sh you later run the `python > setup.py build`? > If so, I'd suggest adding `include /socorro_revision.txt` to the MANIFEST.in > file inside the socorro package so that when packaged, it becomes part of > the build. > > For example, configman extracts its version number from a `version.txt` file > that sits inside the package root (not the project root!) and it gets > included in the tarball predictably thanks to this line > https://github.com/mozilla/configman/blob/ > 7df78c569ec93d37443297b256d7e14fb7a6a582/MANIFEST.in#L4 This example is very helpful, thanks! I have a branch I've started working on for this.
Assignee: nobody → rhelmer
Status: NEW → ASSIGNED
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/22316fc15ff70de02330f99f920116e647781abb fix bug 1161812 - put revision files in root of package https://github.com/mozilla/socorro/commit/969693d5a4ff67617e3941fb8f6e2adc55c62e79 Merge pull request #2790 from rhelmer/bug1161812-server-status-busted-package fix bug 1161812 - put revision files in root of package
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/428be3a45ed945da7edbde08252a04610fbbb43b bug 1161812 - copy files to place PHX deploy expects, since that is not running out of the package https://github.com/mozilla/socorro/commit/1cbb1fd30fa3f34467673bbfc7ac8c9f0dd644a8 Merge pull request #2793 from rhelmer/bug1161812-server-status-busted-package bug 1161812 - copy files to place PHX deploy expects, since that is n…
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/5a3c0d4ef413f5cdd473b7dcfc38f21e9ab169a9 bug 1161812 - copy into socorro package https://github.com/mozilla/socorro/commit/b3114a7e4ad31c7c119114a678adb2c7ac6a147c Merge pull request #2795 from rhelmer/bug1161812-server-status-busted-package bug 1161812 - copy into socorro package
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/9d46495ece04213dea98546bbbceff9007d4f105 bug 1161812 - install to virtualenv before it gets copied into build dir https://github.com/mozilla/socorro/commit/4da50bd2e5afd7b44524f1c4e58bbcf7ae8110aa Merge pull request #2799 from rhelmer/bug1161812-server-status-busted-package bug 1161812 - install to virtualenv before it gets copied into build dir
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/64416e25875cc82c5a5e008ecb98e928ea4a599d bug 1161812 - manipulate revision files before copying into build https://github.com/mozilla/socorro/commit/0ad9e46284d555f81b6205a6618fb44bdafa333c Merge pull request #2800 from rhelmer/bug1161812-server-status-busted-package bug 1161812 - manipulate revision files before copying into build
Commits pushed to master at https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/d49f834fac9ac6a493d5bf2945ab2277bc8badf9 bug 1161812 - build_dir must be modified last https://github.com/mozilla/socorro/commit/4da3e7b3ac45060595d54088e0f69e36955b491b Merge pull request #2801 from rhelmer/bug1161812-server-status-busted-package bug 1161812 - build_dir must be modified last
Product: Socorro → Socorro Graveyard
You need to log in before you can comment on or make changes to this bug.