1032491 - Move old CentOS repos to S3

Status: RESOLVED DUPLICATE of bug 1022763

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Our mock builds run against a "frozen" copy of the CentOS repositories, and we want to host that somewhere other than puppetagain.  We're going to experiment with putting that in S3.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

The repos to mirror:

baseurl=http://repos/repos/yum/mirrors/centos/6/latest/os/$arch
baseurl=http://repos/repos/yum/mirrors/centos/6/latest/updates/$arch
baseurl=http://repos/repos/yum/releng/public/CentOS/6/$arch
baseurl=http://repos/repos/yum/releng/public/CentOS/6/noarch

Notes from Catlee:

----

We could have buckets per region. We're already doing that for Try.
Machine configuration becomes a bit more complex, but should be
manageable. (extreme tangent - I've been thinking we need some kind of
centralized configuration service to handle things like this...kind of
like slavealloc, but for host/bucket/other names)

Use reduced redundancy pricing to limit the storage costs.

Transfer costs are "free" within the same region I think.

tl;dr - we should definitely give it a shot. Nick's been using some s3
rsync thing for the inbound-archive, and it's been working pretty well.
https://pypi.python.org/pypi/boto_rsync/0.8.1 I think?

----

The repos are tiny - just a few gigs - so I don't think reduced redundancy is a win (since it brings an increased likelihood of data loss, and there's no way to re-create these repos)

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

How can I get credentials for this upload?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Whoops, I've been handling this in the parent bug.