Open Bug 1428466 (hgwebaws) Opened 7 years ago Updated 4 years ago

[meta] Establish mirrors of hg.mozilla.org in AWS and use them from Firefox CI

Tracking

(Not tracked)

Status:

REOPENED

People

(Reporter: gps, Assigned: sheehan)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

(Keywords: leave-open, meta)

User Story

Goals of this project:

* Improve disaster recovery story of hg.mo by increasing the number of datacenters where the service runs from
* Improve the reliability of CI by having CI consume a *private* instance of the hg.mo service (instead of a shared public service)
* Improve performance of CI by reducing latency and increasing throughput to service
* Allow us to deploy remotefilelog (enables shallow and narrow clones), which will make VCS operations in CI faster. remotefilelog will likely require memcached (or similar) memory-backed key-value store to achieve peak performance.

There are several components to this project. Some of which aren't fully flushed out:

* Network connectivity and trust relationship between hosts in AWS and the hgssh leader server (currently in SCL3)
* Hostname, DNS, and x509 certificate management (i.e. do we try to use hg.mozilla.org everywhere, per-region hostnames, unified "service" hostname for CI, etc)
* Integration between replication lag and pulse/sns message delivery
* How to leverage snapshots to facilitate efficient machine provisioning
* Which repositories to replicate. e.g. do we need to replicate user repos
* Which AWS account this should be based in (possibly influenced by access control requirements of TaskCluster)
* How machine provisioning / configuration should work (Ansible, Puppet, Terraform, CloudFormation, etc)
* remotefilelog stability. Currently a Facebook extension. Plans to integrate into core Mercurial. Unsure of backwards compatibility story. Should Mozilla help with the upstreaming?
* Interaction between this project and moving hg.mo out of SCL3. e.g. could we / should we consider moving *public* hgweb service from Mozilla datacenter to AWS?

Tentative milestones:

1. Establish an EC2 instance that can replicate *some* changes from the Kafka-based replication system (running in SCL3), proving the viability of cross-datacenter replication.
2. Productionize hgweb mirror service in AWS
3. Switch Firefox CI to use local AWS region service instead of public hg.mo service
4. Deploy remotefilelog and enable for Firefox CI

Gregory Szorc [:gps]

Reporter

Description

•

7 years ago

We want to establish read-only mirrors of hg.mozilla.org in AWS and use those mirrors in Firefox CI. More details in the user story.

Gregory Szorc [:gps]

Reporter