1293682 - Deploy vendor Bundle to all servers

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Dylan Hardison [:dylan] (he/him)]

After we figure out how we're deploying this in bug 1293321, we need to deploy to the dev servers first. Following that, we should verify that they are loading dependencies from the $BUGZILLA_DIR/local directory by removing a system perl module that Bugzilla uses but isn't needed for anything else.

I'll outline the plan in the user story as soon as bug 1293321 is done.

Comment 1

[Dylan Hardison [:dylan] (he/him)]

Update the push scripts to populate /opt/bmo/local.

Something like:

wget http://s3.amazonaws.com/moz-devservices-bmocartons/bmo/vendor.tar.gz
tar -C /opt -xvf vendor.tar.gz bmo/local/

should work.

Comment 2

[Dylan Hardison [:dylan] (he/him)]

the deployment script will also need to ln -s /opt/bmo/local local in the bugzilla dir. this symlink is ignored by git

Comment 3

[Kendall Libby [:fubar] (he/him)]

I'd have a poke around the deploy scripts again and I'd like to propose a change.: instead of using /opt/bmo/local/, we use /data/www/bmo/local/. 

The update-bugzilla-* scripts use parts of the old webops deploy tool, chief; in particular, /data/bugzilla{,-dev,-stage}/deploy. It copies stuff from /data/*/src/ to /data/*/www/, and then instructs each node to rsync it from the admin node to /data/www/*.

So if we extract the vendor tarball into /data/bugzilla*/src/bmo/ on the admin node, the deploy stuff will automatically push it out to the nodes. 

The symlink would also need to change to "../bmo/local", since the admin node doesn't have /data/www and it needs to run stuff like checkperms against the correct vendor tarball.

I think we need to have consensus on how we're managing cartons across the various infrastructures, though. I see you're using 'bmo' on dev, and it sounds like you expect to use that for dev+stage+prod. What happens when we need to do a slow roll out of a change? Prod won't break, generally, because it's a manual process, but I see room for error even there. 

I had envisioned having separate tarballs for dev/stage/prod in scl3, and aws-prod, but perhaps that's too much? What would be a good middle ground that's not a lot of extra work for you, but also minimizes risk to the infra?

Comment 4

[Dylan Hardison [:dylan] (he/him)]

(In reply to Kendall Libby [:fubar] from comment #3)

> The symlink would also need to change to "../bmo/local", since the admin
> node doesn't have /data/www and it needs to run stuff like checkperms
> against the correct vendor tarball.

A relative symlink won't work because we have no assurance what the cwd will be at the time
when perl goes to load modules ( some code changes the cwd, for instance ).


> I think we need to have consensus on how we're managing cartons across the
> various infrastructures, though. I see you're using 'bmo' on dev, and it
> sounds like you expect to use that for dev+stage+prod. What happens when we
> need to do a slow roll out of a change? Prod won't break, generally, because
> it's a manual process, but I see room for error even there. 

We could version them using the short hash of the git commit on the carton-bundles repo.
If we *do* version control the symlink we'd have that assurance of consistency from the developers' machine,
to CI, to development, staging and production and the ability to do a rollback (provided we don't delete where-ever the symlink points to).

> I had envisioned having separate tarballs for dev/stage/prod in scl3, and
> aws-prod, but perhaps that's too much? What would be a good middle ground
> that's not a lot of extra work for you, but also minimizes risk to the infra?

How does the versioning plan sound? It is almost like vendoring all the dependencies into the repo without the repo bloat.

Comment 5

[Dylan Hardison [:dylan] (he/him)]

> The symlink would also need to change to "../bmo/local", since the admin
> node doesn't have /data/www and it needs to run stuff like checkperms
> against the correct vendor tarball.
> 

Since we can't do relative, can the admin node just have a /data/www too?

Comment 6

[Kendall Libby [:fubar] (he/him)]

Yes, but...

It can't be *just* "/data/www" because then on the admin node there's only ever one unpack of the vendor tarball. And the admin node needs to run various cron jobs for each environment. If we don't have separate unpacks we will someday run into a problem.

The simple, if ugly, solution is to just drop it down a level using the "cluster" name: /data/www/bugzilla{,-dev,-stage}/

On the admin node it becomes a chain of links: local -> /data/www/$cluster/bmo/local, and /data/www/$cluster -> /data/$cluster/src/$cluster. But the web nodes are cleaner: local -> /data/www/bugzilla-dev/bmo/local/.

Puppet's all set to enforce the links on the admin node for dev/stage/prod, and the dev update script has been sorted to do the rest. Someone go push to dev and let's see what breaks! ;-)

Comment 7

[Dylan Hardison [:dylan] (he/him)]

This seems to work without a hitch. Will this work in production/stage too? Is it safe for me to push the new code?

Comment 8

[Kendall Libby [:fubar] (he/him)]

I have not applied the same changes to the staging or production scripts, but I can do that on Monday.

Comment 9

[Kendall Libby [:fubar] (he/him)]

applied to prod and staging.