1375551 - circle ci runs tests using wrong code somehow

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We added Circle CI to our continuous integration and use it to build docker images for our code and it'll be used to deploy to the upcoming Socorro infrastructure.

Circle CI runs our unittests using their docker/docker-compose environment. I thought that was working fine, but it's clear it's not.

PR https://github.com/mozilla-services/socorro/pull/3826 has this failure:

"""
==================================== ERRORS ====================================
 ERROR collecting socorro/unittest/middleware/test_middleware_app.py::socorro/unittest/middleware/test_middleware_app::py 
ImportError while importing test module '/app/socorro/unittest/middleware/test_middleware_app.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
socorro/unittest/middleware/test_middleware_app.py:28: in <module>
    from socorro.middleware import middleware_app
socorro/middleware/middleware_app.py:16: in <module>
    import web
E   ImportError: No module named web
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
"""

That file is removed in that PR, so there's no reason for the tests to fail on this.

PR https://github.com/mozilla-services/socorro/pull/3808 also had the same problem and has some commentary in the PR comments on what I did to attempt to resolve it.

This bug covers figuring this out and fixing it. Until we fix it, we shouldn't consider Circle CI's stamp-or-approval as valid.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think this comment sets up the problem specifics nicely:

https://github.com/mozilla-services/socorro/pull/3808#issuecomment-307540077

Trying to figure out how the git checkout is wrong is probably a good starting point.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think I'm going to create a new repository and hook it up to Circle and see if I can reproduce this rather than futz with this in the Socorro-space.

Grabbing this to work on now because it's bad.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 3829: fix circle ci curiosity

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

With Miles' help, I think we figured it out. The current theory is this:

1. Circle spins up the container it uses for Socorro things.
2. Circle goes to check out Socorro, but since "git" isn't installed inside the container, Circle does a git checkout on the host, then does a docker cp to move the results into the container.

Thus at this point, there's a /socorro in the container that's *NOT* a git repository (e.g. there's no /socorro/.git/ in the container) and the files get into the container via a docker cp.

docker cp only copies files one way--it doesn't sync two directories. So if file ABC exists in the destination, but not the source, it doesn't get removed from the destination.

The fix is to install the git package into the container. Then when Circle goes to check out Socorro, it's doing "git checkout xyz" *inside* the container which means that /socorro is a git repository and it also will remove files removed in the branch commits.

Pretty sure that's what's going on. I say that because I fixed a similar issue with "make dockertest" wherein I needed to change it to remove all the files in the container and then do the docker cp.

We'll know if this isn't true when another PR like the one Adrian did comes along. Probably another code removal PR.

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/664bc81c5dda6051badbbbbfaa8a48cfb600bee6
Fixes bug 1375551 - install git into circle environment (#3829)

Ok, so I think what's going on is that our circle configuration downloads an
Ubuntu 14.04 image, installs some packages in it, and then proceeds with the
steps.

On the "checkout:" step, Circle notices that "git" isn't in the container, so
Circle "helpfully" does a git checkout elsehwere, then does a docker cp into the
container, then proceeds with things.

The problem with that is that the container probably has contents from a
previous run. So any files deleted in a PR don't get removed from the container
because docker cp only copies files--it's not syncing files.

The fix here is to install git as a package in the container. Then when it does
the "checkout:" step, it's using git in the container and that will update the
working tree based on the commit correctly and there's no docker cp involved.

Pretty sure that's what's going on. If we bump into this problem again, we can
shake our head in sadness and throw this theory out the window. If that happens,
I suggest we all get religion because I claim the set of possible theories left
will be very small and mostly in the mystical/cosmic domain.