Support fresh server deploys with Ansible

RESOLVED FIXED

Status

Developer Services
Mercurial: hg.mozilla.org
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: gps, Assigned: gps)

Tracking

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(16 attachments)

40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
40 bytes, text/x-review-board-request
fubar
: review+
Details | Review
(Assignee)

Description

2 years ago
hgweb1 was recently re-imaged. As part of this, I'm finding all the pieces of our Ansible / Puppet that are incomplete and require manual intervention to configure a server for full operation. I'm preparing a ton of commits to fill in the gaps.
(Assignee)

Comment 1

2 years ago
Created attachment 8641402 [details]
MozReview Request: ansible/hg-web: clean up package dependencies (bug 1189449); r=fubar

ansible/hg-web: clean up package dependencies (bug 1189449); r?fubar

httpd should be explicitly listed, even though it seems to get pulled in
by httpd-devel.

We don't need the mod_wsgi package because we build mod_wsgi from source
and install it into the virtualenv. (Well, we don't install it from
source - Python's packaging installs it from source.)

The python-* packages aren't necessary, as we run things out of a
virtualenv now and the virtualenv has all the dependencies.
Attachment #8641402 - Flags: review?(klibby)
(Assignee)

Comment 2

2 years ago
Created attachment 8641403 [details]
MozReview Request: ansible/hg-web: move and refactor mirror-pull to hg-web (bug 1189449); r=fubar

ansible/hg-web: move and refactor mirror-pull to hg-web (bug 1189449); r?fubar

The mirror-pull script is used to coordinate replication from master to
mirrors. We had a copy of the script already hooked up to the
docker-hg-web role, but it was done so very hackily. Basically we took
the .erb file from Puppet and checked it in as-is. We then modified the
file in Docker entrypoint script for that container. It was hacky.

We move mirror-pull to the hg-web role. Some of the variable
placeholders have been replaced with hard-coded values because they
don't vary in production nor in Docker. What template syntax remains we
convert to Jinja2. Default values for these variables have been added to
the hg-web role.

The entrypoint hackery in docker-hg-web still remains. It can be cleaned
up later.
Attachment #8641403 - Flags: review?(klibby)
(Assignee)

Comment 3

2 years ago
Created attachment 8641404 [details]
MozReview Request: ansible/hg-web: move lockfile and repo-group to hg-web role (bug 1189449); r=fubar

ansible/hg-web: move lockfile and repo-group to hg-web role (bug 1189449); r?fubar

I confirmed the contents of these currently matches what is in Puppet.
This brings us one step closer to parity with the "hg" Puppet module.

FWIW, I'm not sure if these scripts are still used. But better safe than
sorry.
Attachment #8641404 - Flags: review?(klibby)
(Assignee)

Comment 4

2 years ago
Created attachment 8641405 [details]
MozReview Request: ansible/hg-web: make all httpd prefork settings configurable (bug 1189449); r=fubar

ansible/hg-web: make all httpd prefork settings configurable (bug 1189449); r?fubar

It seems like a good idea to have knobs for all of these.

While I was here, I also removed the IfModule block around prefork
settings and the worker mpm settings completely. My thinking here is
that the server should *always* be in prefork mode and we want it to
fail fast if it is accidentally started with a different mpm.
Attachment #8641405 - Flags: review?(klibby)
(Assignee)

Comment 5

2 years ago
Created attachment 8641406 [details]
MozReview Request: ansible/docker-hg-web: don't update httpd configs in entrypoint script (bug 1189449); r=fubar

ansible/docker-hg-web: don't update httpd configs in entrypoint script (bug 1189449); r?fubar

We have Ansible variables. Let's use them.
Attachment #8641406 - Flags: review?(klibby)
(Assignee)

Comment 6

2 years ago
Created attachment 8641407 [details]
MozReview Request: ansible/hg-web: install other system packages (bug 1189449); r=fubar

ansible/hg-web: install other system packages (bug 1189449); r?fubar

docker-hg-web was installing these packages. While they are likely
already installed on servers we provision, it doesn't hurt to list them
as explicit package requirements. One less one-off in the docker-hg-web
role.
Attachment #8641407 - Flags: review?(klibby)
(Assignee)

Comment 7

2 years ago
Created attachment 8641408 [details]
MozReview Request: ansible/hg-web: install WSGI files from version-control-tools (bug 1189449); r=fubar

ansible/hg-web: install WSGI files from version-control-tools (bug 1189449); r?fubar

The content of /repo/hg/webroot_wsgi was previously managed by hand.
There was little record keeping outside of bugs. And the process for
synchronizing changes across hosts was manual.

This commit does 2 things. First, we import the existing
/repo/hg/webroot_wsgi directory from its canonical home - hgssh1. The
contents of the users/ directory has been excluded from the import
because it is generated by a CRON job. Second, we add an Ansible task
for deploying these files. We use rsync with a "protect" rule to prevent
the contents of the users/ directory from being deleted.
Attachment #8641408 - Flags: review?(klibby)
(Assignee)

Comment 8

2 years ago
Created attachment 8641409 [details]
MozReview Request: ansible/hg-web: ensure httpd service is started (bug 1189449); r=fubar

ansible/hg-web: ensure httpd service is started (bug 1189449); r?fubar

On a freshly imaged machine, httpd didn't start after Ansible installed
it. Add support for doing that.
Attachment #8641409 - Flags: review?(klibby)
(Assignee)

Comment 9

2 years ago
Created attachment 8641410 [details]
MozReview Request: ansible/hg: install mirror SSH key in hg-web role (bug 1189449); r=fubar

ansible/hg: install mirror SSH key in hg-web role (bug 1189449); r?fubar

Freshly imaged machines need an SSH key installed for mirroring. While
we should ideally be fetching this from a secret store and using a
separate key per host, that's not how we do things today.

This commit establishes a hack of sorts to capture the SSH private key
from the hgssh1 host into a variable which is later written as part of
the hg-web role.
Attachment #8641410 - Flags: review?(klibby)
(Assignee)

Comment 10

2 years ago
Created attachment 8641411 [details]
MozReview Request: ansible/hg-web: install SSH known_hosts file for hg user (bug 1189449); r=fubar

ansible/hg-web: install SSH known_hosts file for hg user (bug 1189449); r?fubar

The ~/.ssh/known_hosts file for the hg user needs to contain entries for
the mirror host before SSH will allow connections. Automatically
configure the file with the host key of the mirror server.
Attachment #8641411 - Flags: review?(klibby)
(Assignee)

Comment 11

2 years ago
Created attachment 8641412 [details]
MozReview Request: ansible/hg-web: rename requirements.txt to requirements-hgweb.txt (bug 1189449); r=fubar

ansible/hg-web: rename requirements.txt to requirements-hgweb.txt (bug 1189449); r?fubar

We'll be creating a new virtualenv. Make room by giving
"requirements.txt" a more specific name.
Attachment #8641412 - Flags: review?(klibby)
(Assignee)

Comment 12

2 years ago
Created attachment 8641413 [details]
MozReview Request: ansible/hg-web: create a virtualenv for tools (bug 1189449); r=fubar

ansible/hg-web: create a virtualenv for tools (bug 1189449); r?fubar

We want to run more of our Python tooling with Python 2.7 and with
access to 3rd party Python libraries. This requires the use of a
virtualenv. Since the hgweb virtualenv is specialized and not
appropriate for general use, we create a new virtualenv to be used for
not hgweb things. Its use will become more apparent in subsequent
commits.
Attachment #8641413 - Flags: review?(klibby)
(Assignee)

Comment 13

2 years ago
Created attachment 8641414 [details]
MozReview Request: scripts: script to create a manifest of available repositories (bug 1189449); r=fubar

scripts: script to create a manifest of available repositories (bug 1189449); r?fubar

In order to efficiently bootstrap a new server, we want to clone several
repositories concurrently. In order to do this properly, we need to
respect certain repository properties, such as generaldelta.

We create a script that can produce a manifest of all known repositories
along with basic metadata. We'll use the output of this script to power
a tool for mass cloning/pulling all known repositories.
Attachment #8641414 - Flags: review?(klibby)
(Assignee)

Comment 14

2 years ago
Created attachment 8641415 [details]
MozReview Request: ansible/hg-web: add futures to tools virtualenv (bug 1189449); r=fubar

ansible/hg-web: add futures to tools virtualenv (bug 1189449); r?fubar

This is a useful library for doing things concurrently.
Attachment #8641415 - Flags: review?(klibby)
(Assignee)

Comment 15

2 years ago
Created attachment 8641416 [details]
MozReview Request: scripts: add script to efficiently sync hg repos from a manifest (bug 1189449); r=fubar

scripts: add script to efficiently sync hg repos from a manifest (bug 1189449); r?fubar

Currently, we have no effective way for mass cloning repositories to
machines. Previously, the most effective method we had was to rsync from
an existing server. However, rsync is not ideal from a Mercurial
perspective because it preserve the internal state of the repository,
including any ancient Mercurial settings ("requirements" in Mercurial
parlance) that were in effect when the repository was created for an
older Mercurial version. `hg clone`/`hg init` create new repositories
using the latest greatest feature set and `hg clone`/`hg pull` will auto
convert repository data to the newest/best version.

We establish a script that mass clones repositories. Give it a manifest
of repositories, a destination directory, and a base URL and it does the
rest. It will automatically use all available CPU cores for cloning. It
will preserve generaldelta on repositories.

The script integrates with the "mirror-pull" tool to synchronize hgrc
files. It should arguably be using mirror-pull for all cloning. However,
my intent is to eventually kill mirror-pull (at least in its current
form) because most of its functionality is not necessary, as "hg clone"
and/or "hg pull" or often sufficient. The script is overly complicated
and does things which aren't necessary, such as obtaining a replication
lock (Mercurial already has locking built in).
Attachment #8641416 - Flags: review?(klibby)

Updated

2 years ago
Attachment #8641402 - Flags: review?(klibby) → review+
Comment on attachment 8641402 [details]
MozReview Request: ansible/hg-web: clean up package dependencies (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14485/#review13139

Ship It!

Updated

2 years ago
Attachment #8641403 - Flags: review?(klibby) → review+
Comment on attachment 8641403 [details]
MozReview Request: ansible/hg-web: move and refactor mirror-pull to hg-web (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14487/#review13143

Ship It!

Updated

2 years ago
Attachment #8641404 - Flags: review?(klibby) → review+
Comment on attachment 8641404 [details]
MozReview Request: ansible/hg-web: move lockfile and repo-group to hg-web role (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14489/#review13145

repo-group is pointless on hgweb nodes, because we're not preserving group ownership when we mirror; they're all hg:hg. we do maintain it on ssh://hg.m.o/, though it's still largely unknown.

I have NFC what lockfile is/was used by.
Comment on attachment 8641405 [details]
MozReview Request: ansible/hg-web: make all httpd prefork settings configurable (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14491/#review13147

Ship It!
Attachment #8641405 - Flags: review?(klibby) → review+
Comment on attachment 8641406 [details]
MozReview Request: ansible/docker-hg-web: don't update httpd configs in entrypoint script (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14493/#review13149

Ship It!
Attachment #8641406 - Flags: review?(klibby) → review+
Comment on attachment 8641407 [details]
MozReview Request: ansible/hg-web: install other system packages (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14495/#review13151

::: ansible/roles/hg-web/tasks/main.yml:55
(Diff revision 1)
> +    - syslog

replace with rsyslog.

syslog isn't used on hgweb nodes, nor does it seem to be available from any of the repos normally configured.

related: we're also installing /etc/rsyslog.d/hg.conf via puppet on the hgweb nodes to log local2.* to /var/log/hg.log
Attachment #8641407 - Flags: review?(klibby)
Comment on attachment 8641408 [details]
MozReview Request: ansible/hg-web: install WSGI files from version-control-tools (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14497/#review13155

::: hgwsgi/hgweb_servers.txt:1
(Diff revision 1)
> +hgwebservers	dm-vcview03

pretty sure this is ancient history and can be removed.
Attachment #8641408 - Flags: review?(klibby) → review+
Comment on attachment 8641409 [details]
MozReview Request: ansible/hg-web: ensure httpd service is started (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14499/#review13161

Ship It!
Attachment #8641409 - Flags: review?(klibby) → review+

Updated

2 years ago
Attachment #8641410 - Flags: review?(klibby) → review+
Comment on attachment 8641410 [details]
MozReview Request: ansible/hg: install mirror SSH key in hg-web role (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14501/#review13167

Ship It!
Comment on attachment 8641411 [details]
MozReview Request: ansible/hg-web: install SSH known_hosts file for hg user (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14503/#review13173

Ship It!
Attachment #8641411 - Flags: review?(klibby) → review+

Updated

2 years ago
Attachment #8641412 - Flags: review?(klibby) → review+
Comment on attachment 8641412 [details]
MozReview Request: ansible/hg-web: rename requirements.txt to requirements-hgweb.txt (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14505/#review13175

Ship It!

Updated

2 years ago
Attachment #8641413 - Flags: review?(klibby) → review+
Comment on attachment 8641413 [details]
MozReview Request: ansible/hg-web: create a virtualenv for tools (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14507/#review13177

Ship It!
Comment on attachment 8641416 [details]
MozReview Request: scripts: add script to efficiently sync hg repos from a manifest (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14513/#review13179

Ship It!
Attachment #8641416 - Flags: review?(klibby) → review+
Comment on attachment 8641414 [details]
MozReview Request: scripts: script to create a manifest of available repositories (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14509/#review13181

Ship It!
Attachment #8641414 - Flags: review?(klibby) → review+
Comment on attachment 8641415 [details]
MozReview Request: ansible/hg-web: add futures to tools virtualenv (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14511/#review13183

Ship It!
Attachment #8641415 - Flags: review?(klibby) → review+
(Assignee)

Comment 31

2 years ago
https://reviewboard.mozilla.org/r/14489/#review13145

Well, they are already in version control. I'll clean this up later whenever I take a machete to the replication code.
(Assignee)

Comment 32

2 years ago
https://reviewboard.mozilla.org/r/14495/#review13151

> replace with rsyslog.
> 
> syslog isn't used on hgweb nodes, nor does it seem to be available from any of the repos normally configured.
> 
> related: we're also installing /etc/rsyslog.d/hg.conf via puppet on the hgweb nodes to log local2.* to /var/log/hg.log

Good catch! I'll submit a new commit with this addition.
(Assignee)

Comment 33

2 years ago
Comment on attachment 8641402 [details]
MozReview Request: ansible/hg-web: clean up package dependencies (bug 1189449); r=fubar

ansible/hg-web: clean up package dependencies (bug 1189449); r=fubar

httpd should be explicitly listed, even though it seems to get pulled in
by httpd-devel.

We don't need the mod_wsgi package because we build mod_wsgi from source
and install it into the virtualenv. (Well, we don't install it from
source - Python's packaging installs it from source.)

The python-* packages aren't necessary, as we run things out of a
virtualenv now and the virtualenv has all the dependencies.
Attachment #8641402 - Attachment description: MozReview Request: ansible/hg-web: clean up package dependencies (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: clean up package dependencies (bug 1189449); r=fubar
(Assignee)

Updated

2 years ago
Attachment #8641403 - Attachment description: MozReview Request: ansible/hg-web: move and refactor mirror-pull to hg-web (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: move and refactor mirror-pull to hg-web (bug 1189449); r=fubar
(Assignee)

Comment 34

2 years ago
Comment on attachment 8641403 [details]
MozReview Request: ansible/hg-web: move and refactor mirror-pull to hg-web (bug 1189449); r=fubar

ansible/hg-web: move and refactor mirror-pull to hg-web (bug 1189449); r=fubar

The mirror-pull script is used to coordinate replication from master to
mirrors. We had a copy of the script already hooked up to the
docker-hg-web role, but it was done so very hackily. Basically we took
the .erb file from Puppet and checked it in as-is. We then modified the
file in Docker entrypoint script for that container. It was hacky.

We move mirror-pull to the hg-web role. Some of the variable
placeholders have been replaced with hard-coded values because they
don't vary in production nor in Docker. What template syntax remains we
convert to Jinja2. Default values for these variables have been added to
the hg-web role.

The entrypoint hackery in docker-hg-web still remains. It can be cleaned
up later.
(Assignee)

Comment 35

2 years ago
Comment on attachment 8641404 [details]
MozReview Request: ansible/hg-web: move lockfile and repo-group to hg-web role (bug 1189449); r=fubar

ansible/hg-web: move lockfile and repo-group to hg-web role (bug 1189449); r=fubar

I confirmed the contents of these currently matches what is in Puppet.
This brings us one step closer to parity with the "hg" Puppet module.

FWIW, I'm not sure if these scripts are still used. But better safe than
sorry.
Attachment #8641404 - Attachment description: MozReview Request: ansible/hg-web: move lockfile and repo-group to hg-web role (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: move lockfile and repo-group to hg-web role (bug 1189449); r=fubar
(Assignee)

Comment 36

2 years ago
Comment on attachment 8641405 [details]
MozReview Request: ansible/hg-web: make all httpd prefork settings configurable (bug 1189449); r=fubar

ansible/hg-web: make all httpd prefork settings configurable (bug 1189449); r=fubar

It seems like a good idea to have knobs for all of these.

While I was here, I also removed the IfModule block around prefork
settings and the worker mpm settings completely. My thinking here is
that the server should *always* be in prefork mode and we want it to
fail fast if it is accidentally started with a different mpm.
Attachment #8641405 - Attachment description: MozReview Request: ansible/hg-web: make all httpd prefork settings configurable (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: make all httpd prefork settings configurable (bug 1189449); r=fubar
(Assignee)

Comment 37

2 years ago
Comment on attachment 8641406 [details]
MozReview Request: ansible/docker-hg-web: don't update httpd configs in entrypoint script (bug 1189449); r=fubar

ansible/docker-hg-web: don't update httpd configs in entrypoint script (bug 1189449); r=fubar

We have Ansible variables. Let's use them.
Attachment #8641406 - Attachment description: MozReview Request: ansible/docker-hg-web: don't update httpd configs in entrypoint script (bug 1189449); r?fubar → MozReview Request: ansible/docker-hg-web: don't update httpd configs in entrypoint script (bug 1189449); r=fubar
(Assignee)

Comment 38

2 years ago
Comment on attachment 8641407 [details]
MozReview Request: ansible/hg-web: install other system packages (bug 1189449); r=fubar

ansible/hg-web: install other system packages (bug 1189449); r=fubar

docker-hg-web was installing these packages. While they are likely
already installed on servers we provision, it doesn't hurt to list them
as explicit package requirements. One less one-off in the docker-hg-web
role.
Attachment #8641407 - Attachment description: MozReview Request: ansible/hg-web: install other system packages (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: install other system packages (bug 1189449); r=fubar
Attachment #8641407 - Flags: review?(klibby)
(Assignee)

Updated

2 years ago
Attachment #8641408 - Attachment description: MozReview Request: ansible/hg-web: install WSGI files from version-control-tools (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: install WSGI files from version-control-tools (bug 1189449); r=fubar
(Assignee)

Comment 39

2 years ago
Comment on attachment 8641408 [details]
MozReview Request: ansible/hg-web: install WSGI files from version-control-tools (bug 1189449); r=fubar

ansible/hg-web: install WSGI files from version-control-tools (bug 1189449); r=fubar

The content of /repo/hg/webroot_wsgi was previously managed by hand.
There was little record keeping outside of bugs. And the process for
synchronizing changes across hosts was manual.

This commit does 2 things. First, we import the existing
/repo/hg/webroot_wsgi directory from its canonical home - hgssh1. The
contents of the users/ directory has been excluded from the import
because it is generated by a CRON job. Second, we add an Ansible task
for deploying these files. We use rsync with a "protect" rule to prevent
the contents of the users/ directory from being deleted.
(Assignee)

Comment 40

2 years ago
Comment on attachment 8641409 [details]
MozReview Request: ansible/hg-web: ensure httpd service is started (bug 1189449); r=fubar

ansible/hg-web: ensure httpd service is started (bug 1189449); r=fubar

On a freshly imaged machine, httpd didn't start after Ansible installed
it. Add support for doing that.
Attachment #8641409 - Attachment description: MozReview Request: ansible/hg-web: ensure httpd service is started (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: ensure httpd service is started (bug 1189449); r=fubar
(Assignee)

Comment 41

2 years ago
Comment on attachment 8641410 [details]
MozReview Request: ansible/hg: install mirror SSH key in hg-web role (bug 1189449); r=fubar

ansible/hg: install mirror SSH key in hg-web role (bug 1189449); r=fubar

Freshly imaged machines need an SSH key installed for mirroring. While
we should ideally be fetching this from a secret store and using a
separate key per host, that's not how we do things today.

This commit establishes a hack of sorts to capture the SSH private key
from the hgssh1 host into a variable which is later written as part of
the hg-web role.
Attachment #8641410 - Attachment description: MozReview Request: ansible/hg: install mirror SSH key in hg-web role (bug 1189449); r?fubar → MozReview Request: ansible/hg: install mirror SSH key in hg-web role (bug 1189449); r=fubar
(Assignee)

Comment 42

2 years ago
Comment on attachment 8641411 [details]
MozReview Request: ansible/hg-web: install SSH known_hosts file for hg user (bug 1189449); r=fubar

ansible/hg-web: install SSH known_hosts file for hg user (bug 1189449); r=fubar

The ~/.ssh/known_hosts file for the hg user needs to contain entries for
the mirror host before SSH will allow connections. Automatically
configure the file with the host key of the mirror server.
Attachment #8641411 - Attachment description: MozReview Request: ansible/hg-web: install SSH known_hosts file for hg user (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: install SSH known_hosts file for hg user (bug 1189449); r=fubar
(Assignee)

Comment 43

2 years ago
Comment on attachment 8641412 [details]
MozReview Request: ansible/hg-web: rename requirements.txt to requirements-hgweb.txt (bug 1189449); r=fubar

ansible/hg-web: rename requirements.txt to requirements-hgweb.txt (bug 1189449); r=fubar

We'll be creating a new virtualenv. Make room by giving
"requirements.txt" a more specific name.
Attachment #8641412 - Attachment description: MozReview Request: ansible/hg-web: rename requirements.txt to requirements-hgweb.txt (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: rename requirements.txt to requirements-hgweb.txt (bug 1189449); r=fubar
(Assignee)

Updated

2 years ago
Attachment #8641413 - Attachment description: MozReview Request: ansible/hg-web: create a virtualenv for tools (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: create a virtualenv for tools (bug 1189449); r=fubar
(Assignee)

Comment 44

2 years ago
Comment on attachment 8641413 [details]
MozReview Request: ansible/hg-web: create a virtualenv for tools (bug 1189449); r=fubar

ansible/hg-web: create a virtualenv for tools (bug 1189449); r=fubar

We want to run more of our Python tooling with Python 2.7 and with
access to 3rd party Python libraries. This requires the use of a
virtualenv. Since the hgweb virtualenv is specialized and not
appropriate for general use, we create a new virtualenv to be used for
not hgweb things. Its use will become more apparent in subsequent
commits.
(Assignee)

Comment 45

2 years ago
Comment on attachment 8641414 [details]
MozReview Request: scripts: script to create a manifest of available repositories (bug 1189449); r=fubar

scripts: script to create a manifest of available repositories (bug 1189449); r=fubar

In order to efficiently bootstrap a new server, we want to clone several
repositories concurrently. In order to do this properly, we need to
respect certain repository properties, such as generaldelta.

We create a script that can produce a manifest of all known repositories
along with basic metadata. We'll use the output of this script to power
a tool for mass cloning/pulling all known repositories.
Attachment #8641414 - Attachment description: MozReview Request: scripts: script to create a manifest of available repositories (bug 1189449); r?fubar → MozReview Request: scripts: script to create a manifest of available repositories (bug 1189449); r=fubar
(Assignee)

Comment 46

2 years ago
Comment on attachment 8641415 [details]
MozReview Request: ansible/hg-web: add futures to tools virtualenv (bug 1189449); r=fubar

ansible/hg-web: add futures to tools virtualenv (bug 1189449); r=fubar

This is a useful library for doing things concurrently.
Attachment #8641415 - Attachment description: MozReview Request: ansible/hg-web: add futures to tools virtualenv (bug 1189449); r?fubar → MozReview Request: ansible/hg-web: add futures to tools virtualenv (bug 1189449); r=fubar
(Assignee)

Comment 47

2 years ago
Comment on attachment 8641416 [details]
MozReview Request: scripts: add script to efficiently sync hg repos from a manifest (bug 1189449); r=fubar

scripts: add script to efficiently sync hg repos from a manifest (bug 1189449); r=fubar

Currently, we have no effective way for mass cloning repositories to
machines. Previously, the most effective method we had was to rsync from
an existing server. However, rsync is not ideal from a Mercurial
perspective because it preserve the internal state of the repository,
including any ancient Mercurial settings ("requirements" in Mercurial
parlance) that were in effect when the repository was created for an
older Mercurial version. `hg clone`/`hg init` create new repositories
using the latest greatest feature set and `hg clone`/`hg pull` will auto
convert repository data to the newest/best version.

We establish a script that mass clones repositories. Give it a manifest
of repositories, a destination directory, and a base URL and it does the
rest. It will automatically use all available CPU cores for cloning. It
will preserve generaldelta on repositories.

The script integrates with the "mirror-pull" tool to synchronize hgrc
files. It should arguably be using mirror-pull for all cloning. However,
my intent is to eventually kill mirror-pull (at least in its current
form) because most of its functionality is not necessary, as "hg clone"
and/or "hg pull" or often sufficient. The script is overly complicated
and does things which aren't necessary, such as obtaining a replication
lock (Mercurial already has locking built in).
Attachment #8641416 - Attachment description: MozReview Request: scripts: add script to efficiently sync hg repos from a manifest (bug 1189449); r?fubar → MozReview Request: scripts: add script to efficiently sync hg repos from a manifest (bug 1189449); r=fubar
(Assignee)

Comment 48

2 years ago
Created attachment 8641744 [details]
MozReview Request: ansible/hg-web: install rsyslog policy for hg (bug 1189449); r?fubar

ansible/hg-web: install rsyslog policy for hg (bug 1189449); r?fubar
Attachment #8641744 - Flags: review?(klibby)
Comment on attachment 8641744 [details]
MozReview Request: ansible/hg-web: install rsyslog policy for hg (bug 1189449); r?fubar

https://reviewboard.mozilla.org/r/14597/#review13203

Ship It!
Attachment #8641744 - Flags: review?(klibby) → review+
Comment on attachment 8641407 [details]
MozReview Request: ansible/hg-web: install other system packages (bug 1189449); r=fubar

https://reviewboard.mozilla.org/r/14495/#review13205

Ship It!
Attachment #8641407 - Flags: review?(klibby) → review+
(Assignee)

Comment 51

2 years ago
url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/727682187aef808e98ca5ee88b7d9064ec69a837
changeset:  727682187aef808e98ca5ee88b7d9064ec69a837
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:45:00 2015 -0700
description:
ansible/hg-web: clean up package dependencies (bug 1189449); r=fubar

httpd should be explicitly listed, even though it seems to get pulled in
by httpd-devel.

We don't need the mod_wsgi package because we build mod_wsgi from source
and install it into the virtualenv. (Well, we don't install it from
source - Python's packaging installs it from source.)

The python-* packages aren't necessary, as we run things out of a
virtualenv now and the virtualenv has all the dependencies.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/b97c01ad3676340f2f10395d3fbf44f0d5b2e7fe
changeset:  b97c01ad3676340f2f10395d3fbf44f0d5b2e7fe
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:45:14 2015 -0700
description:
ansible/hg-web: move and refactor mirror-pull to hg-web (bug 1189449); r=fubar

The mirror-pull script is used to coordinate replication from master to
mirrors. We had a copy of the script already hooked up to the
docker-hg-web role, but it was done so very hackily. Basically we took
the .erb file from Puppet and checked it in as-is. We then modified the
file in Docker entrypoint script for that container. It was hacky.

We move mirror-pull to the hg-web role. Some of the variable
placeholders have been replaced with hard-coded values because they
don't vary in production nor in Docker. What template syntax remains we
convert to Jinja2. Default values for these variables have been added to
the hg-web role.

The entrypoint hackery in docker-hg-web still remains. It can be cleaned
up later.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/b2aacc18c9c920427f6fa02b04dcee6b6c2cefda
changeset:  b2aacc18c9c920427f6fa02b04dcee6b6c2cefda
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:46:23 2015 -0700
description:
ansible/hg-web: move lockfile and repo-group to hg-web role (bug 1189449); r=fubar

I confirmed the contents of these currently matches what is in Puppet.
This brings us one step closer to parity with the "hg" Puppet module.

FWIW, I'm not sure if these scripts are still used. But better safe than
sorry.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/c27a8761d336cd201321a22948d41e960f00fdff
changeset:  c27a8761d336cd201321a22948d41e960f00fdff
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:46:39 2015 -0700
description:
ansible/hg-web: make all httpd prefork settings configurable (bug 1189449); r=fubar

It seems like a good idea to have knobs for all of these.

While I was here, I also removed the IfModule block around prefork
settings and the worker mpm settings completely. My thinking here is
that the server should *always* be in prefork mode and we want it to
fail fast if it is accidentally started with a different mpm.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/35ffd92a1f9d9fc5b117b809d4090dc888f651df
changeset:  35ffd92a1f9d9fc5b117b809d4090dc888f651df
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:46:53 2015 -0700
description:
ansible/docker-hg-web: don't update httpd configs in entrypoint script (bug 1189449); r=fubar

We have Ansible variables. Let's use them.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/3f2e9146fab4e9a56a91388f95dff5199bc7603b
changeset:  3f2e9146fab4e9a56a91388f95dff5199bc7603b
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:48:10 2015 -0700
description:
ansible/hg-web: install other system packages (bug 1189449); r=fubar

docker-hg-web was installing these packages. While they are likely
already installed on servers we provision, it doesn't hurt to list them
as explicit package requirements. One less one-off in the docker-hg-web
role.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/c7a5d6cac5872fadcbae3d36047cd273dea69c5f
changeset:  c7a5d6cac5872fadcbae3d36047cd273dea69c5f
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:49:41 2015 -0700
description:
ansible/hg-web: install WSGI files from version-control-tools (bug 1189449); r=fubar

The content of /repo/hg/webroot_wsgi was previously managed by hand.
There was little record keeping outside of bugs. And the process for
synchronizing changes across hosts was manual.

This commit does 2 things. First, we import the existing
/repo/hg/webroot_wsgi directory from its canonical home - hgssh1. The
contents of the users/ directory has been excluded from the import
because it is generated by a CRON job. Second, we add an Ansible task
for deploying these files. We use rsync with a "protect" rule to prevent
the contents of the users/ directory from being deleted.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/0e166000f3cabd5b2768194da7180dc775a67ae4
changeset:  0e166000f3cabd5b2768194da7180dc775a67ae4
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 10:08:34 2015 -0700
description:
ansible/hg-web: ensure httpd service is started (bug 1189449); r=fubar

On a freshly imaged machine, httpd didn't start after Ansible installed
it. Add support for doing that.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/1a396014b8fae0f84a327d89dd3594884bb1e725
changeset:  1a396014b8fae0f84a327d89dd3594884bb1e725
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:50:09 2015 -0700
description:
ansible/hg: install mirror SSH key in hg-web role (bug 1189449); r=fubar

Freshly imaged machines need an SSH key installed for mirroring. While
we should ideally be fetching this from a secret store and using a
separate key per host, that's not how we do things today.

This commit establishes a hack of sorts to capture the SSH private key
from the hgssh1 host into a variable which is later written as part of
the hg-web role.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/372b0d077662f051577b967216feb802a543d804
changeset:  372b0d077662f051577b967216feb802a543d804
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:50:34 2015 -0700
description:
ansible/hg-web: install SSH known_hosts file for hg user (bug 1189449); r=fubar

The ~/.ssh/known_hosts file for the hg user needs to contain entries for
the mirror host before SSH will allow connections. Automatically
configure the file with the host key of the mirror server.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/a925e797adade1f06042255229d7037456966814
changeset:  a925e797adade1f06042255229d7037456966814
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:50:48 2015 -0700
description:
ansible/hg-web: rename requirements.txt to requirements-hgweb.txt (bug 1189449); r=fubar

We'll be creating a new virtualenv. Make room by giving
"requirements.txt" a more specific name.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/a06266682b3c7daf26f1a4b2f79eb8b56ff0d00a
changeset:  a06266682b3c7daf26f1a4b2f79eb8b56ff0d00a
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:51:17 2015 -0700
description:
ansible/hg-web: create a virtualenv for tools (bug 1189449); r=fubar

We want to run more of our Python tooling with Python 2.7 and with
access to 3rd party Python libraries. This requires the use of a
virtualenv. Since the hgweb virtualenv is specialized and not
appropriate for general use, we create a new virtualenv to be used for
not hgweb things. Its use will become more apparent in subsequent
commits.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/2a76ba261246b7d810346eab0ae1bab6d724d59c
changeset:  2a76ba261246b7d810346eab0ae1bab6d724d59c
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:51:30 2015 -0700
description:
scripts: script to create a manifest of available repositories (bug 1189449); r=fubar

In order to efficiently bootstrap a new server, we want to clone several
repositories concurrently. In order to do this properly, we need to
respect certain repository properties, such as generaldelta.

We create a script that can produce a manifest of all known repositories
along with basic metadata. We'll use the output of this script to power
a tool for mass cloning/pulling all known repositories.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/9c2e16406c066b61d9c773b396f8b33d1543e2d9
changeset:  9c2e16406c066b61d9c773b396f8b33d1543e2d9
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:51:58 2015 -0700
description:
ansible/hg-web: add futures to tools virtualenv (bug 1189449); r=fubar

This is a useful library for doing things concurrently.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/fd1d6c334aa532c00fc5c98bfa9ad50f8072f944
changeset:  fd1d6c334aa532c00fc5c98bfa9ad50f8072f944
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 09:52:08 2015 -0700
description:
scripts: add script to efficiently sync hg repos from a manifest (bug 1189449); r=fubar

Currently, we have no effective way for mass cloning repositories to
machines. Previously, the most effective method we had was to rsync from
an existing server. However, rsync is not ideal from a Mercurial
perspective because it preserve the internal state of the repository,
including any ancient Mercurial settings ("requirements" in Mercurial
parlance) that were in effect when the repository was created for an
older Mercurial version. `hg clone`/`hg init` create new repositories
using the latest greatest feature set and `hg clone`/`hg pull` will auto
convert repository data to the newest/best version.

We establish a script that mass clones repositories. Give it a manifest
of repositories, a destination directory, and a base URL and it does the
rest. It will automatically use all available CPU cores for cloning. It
will preserve generaldelta on repositories.

The script integrates with the "mirror-pull" tool to synchronize hgrc
files. It should arguably be using mirror-pull for all cloning. However,
my intent is to eventually kill mirror-pull (at least in its current
form) because most of its functionality is not necessary, as "hg clone"
and/or "hg pull" or often sufficient. The script is overly complicated
and does things which aren't necessary, such as obtaining a replication
lock (Mercurial already has locking built in).

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/3b725cad11ec59dd034f68f3dadd1e8fcd4aed65
changeset:  3b725cad11ec59dd034f68f3dadd1e8fcd4aed65
user:       Gregory Szorc <gps@mozilla.com>
date:       Fri Jul 31 10:04:17 2015 -0700
description:
ansible/hg-web: install rsyslog policy for hg (bug 1189449); r=fubar
(Assignee)

Comment 52

2 years ago
And deployed!
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Blocks: 1189932
(Assignee)

Updated

2 years ago
Blocks: 1189938
You need to log in before you can comment on or make changes to this bug.