757285 - Make public mirror of PuppetAgain repos

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Jonathan Griffin (:jgriffin)]

We're planning on using the PuppetAgain repos for installing Python packages on buildbot slaves during build jobs (e.g., talos tests), see bug 701506, bug 757283.  Currently, these packages aren't available publicly, which makes testing and debugging against them hard.

We'd like to mirror these files to a public location to make testing and debugging easier.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I think I will implement this as a Zeus proxy to one or more puppet masters, on a distinct, "public" port.  Then I'll use Apache to prevent access to sensitive materials from that public port.

This will require setting up a puppetmaster in scl3 first, since that's where we'll be serving this from.  The VM's already up, but I'm wrapping up some SSL work before I can initialize it.  So this could be a few days to a week.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

OK, the plan is:

* add a public_mirror => true option to the releng::puppetmaster class
* that causes a number of "trees" to be shared, currently
  - python
  - repos/yum/mirrors
  - repos/yum/releng
  (note that this will not include e.g., repos/yum/private)
* for each tree:
  - set up an rsync module
  - set up an Apache Alias on a distinct port
* Point rsync and http toward this with Zeus on a freshly allocated VIP.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

VIP: puppetagain-zlb.vips.scl3.mozilla.com / 63.245.215.68
CNAME: puppetagain.pub.build.mozilla.org

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

OK, rsync and public http (on port 82) are up.  Both exclude /data/*/private, which exist but are empty right now.  All puppetagain masters serve both.

So, all that remains is to get Zeus configured and pointed toward one of the masters (scl3, since that's where Zeus is).

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Zeus is set up, but disabled pending the flow in the dependent bug.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Rsync is running.  I requested the wrong flow for http, so that's not running yet.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Http's up.