1458150 - Get taskcluster tools deployment to handle dotenv configuration somehow

Status: RESOLVED FIXED

(Taskcluster :: Services, enhancement)

Description

[Brian Stack [:bstack]]

Currently, the tools site runs in staging just fine. It is run by running nginx serving static files out of a build directory.

We're supposed to be able to set various env-vars at runtime for the tools site to be able to set things like application name and root url. Afaict, this gets set by whatever process runs the server during `yarn start`. It does work when running things locally, etc.

The issue is that nginx does not do anything with these env-vars so everything sticks with the default.

Question: Can we solve this in some way that still lets us serve directly via nginx or do we need to run a node process to let dotenv do its magic?

Comment 1

[:Eli Perelman]

Are you running a build with the environment variables in place? For example, if APPLICATION_NAME exists in the environment, then run yarn build?

Comment 2

[Brian Stack [:bstack]]

The issue is that we will do a build only during the build phase (har har har) of cutting a tasklcuster release. Once it has been built and pushed to dockerhub, we never build again. So once you deploy to two different clusters, we'll need to be able to set this at runtime.

Another option is to do a build of tools on every deploy.

I don't see this as an issue with how the tools site is doing things, just with how we've set the container up itself.

Comment 3

[:Eli Perelman]

Yes, tools has to be built with every deployment. Browser code doesn't have the concept of environment variables, so these get injectes into client code at build time; it's not possible for them to get the values at runtime. We can chat more if that isn't clear.

Comment 4

[Brian Stack [:bstack]]

I was reading [0] when I made this. Is it possible to do anything like that? If not, we can also change the deploy process to do a build every time as well like you say.

[0] https://medium.com/comparethemarket/environment-configuration-with-react-nginx-2444e1f11a4c

Comment 5

[:Eli Perelman]

From reading the article, it seems their solution is to generate a file which is generated from the environment and then the client makes an HTTP request for it. I would prefer not to go down this route for caching reasons, and webpack has this nailed down right now. If we start configuring the UI this way, we also lose our ability to tree shake based on the environment. TLDR; I would go this route as a last route, it's not optimal for the UI.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

So that would be:

build (once to create the docker container): `yarn install`
deploy (every time the container is started): `yarn build && nginx start`

The downside to this is, `yarn build` is pretty slow.  So when a container fails, we're going to wait for the total build time before a new container is up and healthy.  It also makes the slug quite a bit larger, as it must include all of the source, webpack, etc.  We also need to scale what should be a small container to have a lot of CPU and RAM for the build process, which then sits idle after the build.

This bothers me, too: "The are far too many possibilities for variance, and therefore failure in the process to ‘recompile’ the code for each environment" -- if the build process fails, the deployment will fail.  Kubernetes will retry and not put a broken Pod into the load balancer, but this could still lead to 5-10 minute wait times for deploying tools.

I can see the benefit of tree-shaking.  For example, a config without auth0 enabled could omit a bunch of auth0, jwt, jwks, etc. packages.

Instead of a separate fetch, we could include the config in `index.html`. Just do a much smaller pre-start operation that sticks the runtime config into `index.html`:
<script>const RUNTIME_CONFIG = { ... stuff from the env vars from kubernetes };</script>
and then use <App runtimeConfig={RUNTIME_CONFIG || defaultConfig} /> to inject that information as props.  This avoids any issues with caching or extra requests.  It loses the benefit of tree-shaking, though.

If we wanted to be fancy, we could have both build-time *and* run-time configuration for things where tree-shaking would help.  This is similar to how server apps like Apache or Postfix are built: you can configure features on and off at build time, and then decide to use them or not at runtime.  Linux distros typically configure most features on at build time, but some users will recompile to disable unused features for security or efficiency reasons.

Comment 7

[:Eli Perelman]

> this could still lead to 5-10 minute wait times for deploying tools

This is based on our current version of Neutrino. The next version improves build times significantly.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

True, but it would still be by far the slowest deployment, since everything else is just running `node src/main server`.

Also, "rollback" isn't as safe if we're re-building on rollback instead of just using already-built data.

Comment 9

[:Eli Perelman]

Correct, we cannot compare compiled vs. non-compiled services and frontend compilation to backend compilation.

> re-building on rollback

So rollback doesn't just use a previous image?

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

> So rollback doesn't just use a previous image?

It does, but what you're suggesting is that that image is a source tree, and that the process of starting an image involves running `neutrino build`.

Comment 11

[:Eli Perelman]

The frontend is not a "runnable" project in production; it is a static collection of files. Nginx is the runnable project, serving these static files to the browser. The frontend is also not runnable from source in production, and requires a compilation step.

> you're suggesting is that that image is a source tree, and that the process of starting an image involves running `neutrino build`

I'm not sure how we jumped to this characterization. The source is used to build an image, and the image should be the output of running `neutrino build`. *Starting* an image shouldn't involve Neutrino, it should be nginx. Am I missing something?

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

It wasn't a big jump from "Yes, tools has to be built with every deployment" :)

Maybe we are interpreting "deployment" differently.  In the context of Taskcluster, there are two phases, "build" and "deploy".  The build phase creates a bunch of docker images which are not site-specific.  The deploy phase then incorporates site-specific configuration and creates the resources to actually run a Taskcluster instance.

So, since the runtime configuration isn't available until the deploy phase, and that configuration serves as input to `neutrino build`, we would have to run `neutrino build` during the deploy phase, which is to say, when each docker container starts.

Comment 13

[:Eli Perelman]

Ah, that's the piece I was missing. So yes, while there are two distinct phases, "builds" can happen in either of them. Tools' build would happen during the "deploy" phase. I wonder if you think we should try to come up with some standardized phase names so we can help avoid this confusion in the future?

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

My task is to modify the build process so that it doesn't run `neutrino build`, and instead just packages up the source directory + node_modules in a docker container.

This will be a little more complicated than I thought: that container needs to have the stuff to run `neutrino build` as well as nginx.  I will probably go with the `node` container and adapt it by apt-get installing nginx.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Not so bad.  The container takes about 20 minutes to start, but it starts.