1443210 - Cut down the number of intermittent failures when building in CI

Status: NEW

(Conduit :: Lando, enhancement, P3)

Description

[Steven MacLeod [:smacleod]]

Our CI builds fail about half the time due to intermittents compiling uwsgi for the container. We should switch to the binary package that comes with alpine to avoid these intermittents (The packaged version is actually newer than what we're compiling anyway).

Comment 1

[Steven MacLeod [:smacleod]]

(In reply to Steven MacLeod [:smacleod] from comment #0)
> Our CI builds fail about half the time due to intermittents compiling uwsgi
> for the container.

So looking into this a little deeper, it actually seems a lot of our intermittents come from attempting to install alpine packages (one of the package index's being temporarily unavailable) or filesystem errors (seeming like the disk is out of space). I'm thinking switching to the uwsgi binary package won't actually make our intermittent failures any less frequent.

Additionally, due to the dockerhub library python3.5 using an older version of alpine than I though, the uwsgi shipped there is older than what we're using.

I think the best way to handle this would be to create our own custom base package that comes with uwsgi compiled or installed already, as well as psycopg2 and pycrypto[1]. This would allow us to avoid installing any build-tools / compilation in our usual builds, cutting down on the intermittent failures by quite a bit.

While I think we should handle this all at some point, I'm no longer sure we should bother with it for the current story. :mcote, thoughts?


[1] There are alpine packages for all of these libraries which we could install through packages rather than compiling but that's only really possible if you base your container on the alpine base image itself rather than python, and use the alpine installed python. Doesn't seem like such a bad idea, the python3 that is in alpine is 3.6.

Comment 2

[Mark Côté [:mcote]]

My first thought is that we can't be the only app to suffer from such problems... are you saying switching base images should fix this?  Is that what other projects have done?  Or are we unique somehow?

Comment 3

[Steven MacLeod [:smacleod]]

(In reply to Mark Côté [:mcote] from comment #2)
> My first thought is that we can't be the only app to suffer from such
> problems... are you saying switching base images should fix this?  Is that
> what other projects have done?  Or are we unique somehow?

We aren't the only app with problems like this, for example :dylan indicated he sees failures all the time for the uploads to dockerhub failing. I think it's just intermittent connectivity issues in Circle CI. All that being said *maybe* moving to circle CI 2.0 would help with things, I'm not sure.

What I'm actually suggesting here though is just having a custom base container with the heavy duty dependencies that we compile built in our own repo, then base lando et al off of that, so we only have to deal with that compilation infrequently. This would mean quicker and hopefully less error prone builds.

The intermittent failures now are annoying, but retriggering the build is easy enough that I'm not sure it's worth putting the time in right now to setup the other base container builds.

Comment 4

[Mark Côté [:mcote]]

Thanks for the explanation.  I will put this on the backlog for now.