815758 - set up staging and production mozpools of pandas

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

One rack (including one mobile-imaging server) for staging, backed by the staging db; the remainder for production.

Both should come from the production racks.  Let's leave the first six chassis as testing/dev for the moment, with hopes they'll eventually be available to move into a production rack.

This will require
* puppet changes to allow servers to run staging or prod
* hacks to the inventory sync so production doesn't "discover" the staging pandas

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Per our meeting, the production pool will be used for both b2g and Android, at releng's discretion.  Mozpool can be used to flexibly reimage machines between those two.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug815758.patch

Comment 3

[Justin Wood (:Callek)]

Comment on attachment 685832 [details] [diff] [review]
bug815758.patch

Review of attachment 685832 [details] [diff] [review]:
-----------------------------------------------------------------

::: manifests/nodes.pp
@@ +73,5 @@
> +    $is_bmm_admin_host = false
> +    include toplevel::server::mozpool
> +}
> +
> +node /mobile-imaging-0(0[2-9]|10)\.p\d+\.releng\.scl1\.mozilla\.com/ {

why using this complicated regex rather than just using /mobile-imaging-\d+\.p\d+\.releng\.scl1\.mozilla\.com/ ?

Puppet node matching will find the *first* that matches, therefore the -001 will match first and then it won't touch this one.

::: modules/mozpool/templates/config.ini.erb
@@ +3,5 @@
>  
>  [inventory]
> +url = <%= scope.lookupvar('::mozpool::settings::inventory_url') %>
> +username = <%= scope.lookupvar('::mozpool::settings::inventory_username') %>
> +password = <%= scope.lookupvar('::mozpool::settings::inventory_password') %>

Are you sure the mozpool::settings::inventory* works, I don't see it defined in settings.pp directly (merely included from config::secrets)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I wasn't aware it picked the first - that's helpful.  The second of those was probably worth an r-!

Fixed and pushed.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

mozpool changes are in
  https://github.com/djmitche/mozpool/pull/7

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

that's landed - now to install it and add the proper config.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug815758.patch

Comment 8

[John Hopkins (:jhopkins)]

Comment on attachment 686168 [details] [diff] [review]
bug815758.patch

passes visual inspection

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

landed; once that's in, I'll run sync scripts on both pools and we'll be set.

Worth noting, I'm treating this as a temporary solution - we certainly don't want a whole rack of "staging pandas" long-term.  Arguably, if we decide to dedicate certain devices to non-production uses, that dedication should be done within a single mozpool cluster, the way we do for slavealloc.  Anyway, that explains why this is kind of a lame implementation.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

OK, everything's synchronized.  We have two pools now.

I do *not* like this - it means they have different pxe configs that we'll need to keep in sync, among other problems.

So, let's plan to use the staging pool for a month or so at most, until we're pretty confident that our tooling's not going to wreak mayhem in production, and then re-merge all of the pandas into the same pool.  As we've discussed, we can use other means to allocate pandas to particular purposes, later.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

This is being reverted in bug 818995