Open Bug 1428466 (hgwebaws) Opened 7 years ago Updated 5 years ago

[meta] Establish mirrors of hg.mozilla.org in AWS and use them from Firefox CI

Categories

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Product:
Developer Services
Component:
Mercurial: hg.mozilla.org
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
REOPENED

People

(Reporter: gps, Assigned: sheehan)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

(Keywords: leave-open, meta)

User Story

Goals of this project:

* Improve disaster recovery story of hg.mo by increasing the number of datacenters where the service runs from
* Improve the reliability of CI by having CI consume a *private* instance of the hg.mo service (instead of a shared public service)
* Improve performance of CI by reducing latency and increasing throughput to service
* Allow us to deploy remotefilelog (enables shallow and narrow clones), which will make VCS operations in CI faster. remotefilelog will likely require memcached (or similar) memory-backed key-value store to achieve peak performance.

There are several components to this project. Some of which aren't fully flushed out:

* Network connectivity and trust relationship between hosts in AWS and the hgssh leader server (currently in SCL3)
* Hostname, DNS, and x509 certificate management (i.e. do we try to use hg.mozilla.org everywhere, per-region hostnames, unified "service" hostname for CI, etc)
* Integration between replication lag and pulse/sns message delivery
* How to leverage snapshots to facilitate efficient machine provisioning
* Which repositories to replicate. e.g. do we need to replicate user repos
* Which AWS account this should be based in (possibly influenced by access control requirements of TaskCluster)
* How machine provisioning / configuration should work (Ansible, Puppet, Terraform, CloudFormation, etc)
* remotefilelog stability. Currently a Facebook extension. Plans to integrate into core Mercurial. Unsure of backwards compatibility story. Should Mozilla help with the upstreaming?
* Interaction between this project and moving hg.mo out of SCL3. e.g. could we / should we consider moving *public* hgweb service from Mozilla datacenter to AWS?

Tentative milestones:

1. Establish an EC2 instance that can replicate *some* changes from the Kafka-based replication system (running in SCL3), proving the viability of cross-datacenter replication.
2. Productionize hgweb mirror service in AWS
3. Switch Firefox CI to use local AWS region service instead of public hg.mo service
4. Deploy remotefilelog and enable for Firefox CI
We want to establish read-only mirrors of hg.mozilla.org in AWS and use those mirrors in Firefox CI. More details in the user story.
Depends on: 1428470
Blocks: 1428470
No longer depends on: 1428470
NI-ing fubar for updates on host based firewall access. :)
Flags: needinfo?(klibby)
Updated local firewall rules to allow ssh; I think everything else should be open.
Flags: needinfo?(klibby)
Assignee: nobody → sheehan
Status: NEW → ASSIGNED
Depends on: 1456998
Alias: hgwebaws
Depends on: 1490403
Depends on: 1491373
Depends on: 1509039
Depends on: 1511058
Depends on: 1512473
Depends on: 1517615
Depends on: 1518394
Depends on: 1518706

Pushed by cosheehan@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/550944957db0
vcsreplicator: add output argument to hgssh bootstrap script
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f260c773c13f
vcsreplicator: logging improvements in bootstrap

Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---
Depends on: 1522935
Depends on: 1525954
Depends on: 1530708
Depends on: 1534463
Depends on: 1535011
Depends on: 1525373
Depends on: 1548302
Depends on: 1550782
See Also: → 1587958
You need to log in before you can comment on or make changes to this bug.