1337626 - EC2 instance for running version control syncing services

Status: RESOLVED WONTFIX

(Developer Services :: Servo VCS Sync, defect)

Description

[Gregory Szorc [:gps]]

Filing a new bug to track standing up an EC2 instance for doing version control syncing stuff, initially for Servo.

Comment 1

[Gregory Szorc [:gps]]

Attachment: ansible: role to manage version control syncing services (bug 1337626);

Introduced is an Ansible role for a machine performing version
control "syncing" functionality. The idea is for all our services
doing version control sync to be defined in this role. Currently
implemented is configuration required for Servo VCS syncing.

TODO:

* Integrate secrets
* Monitoring / Postfix configuration
* Replace some hardcoded uses of my personal account info
* Docs

Review commit: https://reviewboard.mozilla.org/r/110564/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/110564/

Comment 2

[Gregory Szorc [:gps]]

I have a basic Terraform change for this as well. But it is very minimal - all the heavy lifting so far is in the attached Ansible code.

There are several open issues that I need help with:

* Secrets management. We will have SSH keys and other credentials. I'm not sure where we should store secrets. With Ansible, we can put them in a central YAML or JSON file on the server and parse them out.
* SMTP. I have systemd units configured to email on failure. Not sure how we can send email from an EC2 instance. If difficult, we can possibly discard.
* Monitoring. We'll want Nagios or something to alert if systemd services fall over. I'm not sure what the story is for EC2 instances since I've only done monitoring in SCL3.

needinfo dividehex to help with open issues

Comment 3

[Kendall Libby [:fubar] (he/him)]

Is this something that needs to be done/in-place before you disappear? If yes, I'm more than a little worried about supporting it given how fast things have been moving and how little time is left.

Comment 4

[Gregory Szorc [:gps]]

(In reply to Kendall Libby [:fubar] from comment #3)
> Is this something that needs to be done/in-place before you disappear? If
> yes, I'm more than a little worried about supporting it given how fast
> things have been moving and how little time is left.

They are nice to have before I depart but not critical.

I think we care most about knowing if things fail. The others are added convenience.

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8834721 [details]
ansible: role to manage version control syncing services (bug 1337626);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/110564/diff/1-2/

Comment 6

[Gregory Szorc [:gps]]

Attachment: docs: document Servo version control syncing

Review commit: https://reviewboard.mozilla.org/r/111438/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/111438/

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8834721 [details]
ansible: role to manage version control syncing services (bug 1337626);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/110564/diff/2-3/

Comment 8

[:glob ✱]

Comment on attachment 8834721 [details]
ansible: role to manage version control syncing services (bug 1337626);

https://reviewboard.mozilla.org/r/110564/#review112746

::: ansible/hosts:26
(Diff revision 3)
> +# TODO switch to new, production instance.
> +[vcssync]
> +54.202.100.174 ansible_ssh_user=centos ansible_become=yes

reminder to update this.

Comment 9

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8834721 [details]
ansible: role to manage version control syncing services (bug 1337626);

https://reviewboard.mozilla.org/r/110564/#review112766

::: testing/vcttesting/deploy.py:79
(Diff revision 3)
>  
>      return run_playbook('deploy-hgmo', extra_vars=extra,
>                          verbosity=verbosity)
>  
>  
> +def deploy_vcs_sync(verbosity=0):

Maybe consider renaming to deploy_servo_sync to prevent confusion with the other vcs-sync?

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8834721 [details]
ansible: role to manage version control syncing services (bug 1337626);

https://reviewboard.mozilla.org/r/110564/#review112922

I'm going to land this as is. Will be pushing follow-ups all day by the looks of it :/

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a6c0315d939d
ansible: role to manage version control syncing services ; r=glob, fubar

Comment 12

[Amy Rich [:arr] [:arich]]

This proved to be non-functional over the weekend and doesn't have adequate monitoring or a way to escalate to fix it. Reopening until those issues are resolved.

Comment 13

[:glob ✱]

the sync service is no longer running/needed.