circle ci runs tests using wrong code somehow



10 months ago
10 months ago


(Reporter: willkg, Assigned: willkg)


Firefox Tracking Flags

(Not tracked)



(1 attachment)

We added Circle CI to our continuous integration and use it to build docker images for our code and it'll be used to deploy to the upcoming Socorro infrastructure.

Circle CI runs our unittests using their docker/docker-compose environment. I thought that was working fine, but it's clear it's not.

PR has this failure:

==================================== ERRORS ====================================
 ERROR collecting socorro/unittest/middleware/ 
ImportError while importing test module '/app/socorro/unittest/middleware/'.
Hint: make sure your test modules/packages have valid Python names.
socorro/unittest/middleware/ in <module>
    from socorro.middleware import middleware_app
socorro/middleware/ in <module>
    import web
E   ImportError: No module named web
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!

That file is removed in that PR, so there's no reason for the tests to fail on this.

PR also had the same problem and has some commentary in the PR comments on what I did to attempt to resolve it.

This bug covers figuring this out and fixing it. Until we fix it, we shouldn't consider Circle CI's stamp-or-approval as valid.
I think this comment sets up the problem specifics nicely:

Trying to figure out how the git checkout is wrong is probably a good starting point.
I think I'm going to create a new repository and hook it up to Circle and see if I can reproduce this rather than futz with this in the Socorro-space.

Grabbing this to work on now because it's bad.
Assignee: nobody → willkg
Created attachment 8880543 [details] [review]
pr 3829: fix circle ci curiosity
With Miles' help, I think we figured it out. The current theory is this:

1. Circle spins up the container it uses for Socorro things.
2. Circle goes to check out Socorro, but since "git" isn't installed inside the container, Circle does a git checkout on the host, then does a docker cp to move the results into the container.

Thus at this point, there's a /socorro in the container that's *NOT* a git repository (e.g. there's no /socorro/.git/ in the container) and the files get into the container via a docker cp.

docker cp only copies files one way--it doesn't sync two directories. So if file ABC exists in the destination, but not the source, it doesn't get removed from the destination.

The fix is to install the git package into the container. Then when Circle goes to check out Socorro, it's doing "git checkout xyz" *inside* the container which means that /socorro is a git repository and it also will remove files removed in the branch commits.

Pretty sure that's what's going on. I say that because I fixed a similar issue with "make dockertest" wherein I needed to change it to remove all the files in the container and then do the docker cp.

We'll know if this isn't true when another PR like the one Adrian did comes along. Probably another code removal PR.

Comment 5

10 months ago
Commit pushed to master at
Fixes bug 1375551 - install git into circle environment (#3829)

Ok, so I think what's going on is that our circle configuration downloads an
Ubuntu 14.04 image, installs some packages in it, and then proceeds with the

On the "checkout:" step, Circle notices that "git" isn't in the container, so
Circle "helpfully" does a git checkout elsehwere, then does a docker cp into the
container, then proceeds with things.

The problem with that is that the container probably has contents from a
previous run. So any files deleted in a PR don't get removed from the container
because docker cp only copies files--it's not syncing files.

The fix here is to install git as a package in the container. Then when it does
the "checkout:" step, it's using git in the container and that will update the
working tree based on the commit correctly and there's no docker cp involved.

Pretty sure that's what's going on. If we bump into this problem again, we can
shake our head in sadness and throw this theory out the window. If that happens,
I suggest we all get religion because I claim the set of possible theories left
will be very small and mostly in the mystical/cosmic domain.


10 months ago
Last Resolved: 10 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.