773057 - Point slaves to the new slavealloc URL

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

..with PuppetAgain.

This is mainly a concrete goal which will result in setting up the generic toplevel::server stuff.

Slavealloc isn't currently built with puppet (of any sort), so this will be an improvement in reproducibility!  The testing of and transition to the new host should be trivially easy.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug773286.patch

Docs at

https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Modules/shared

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 641523 [details] [diff] [review]
bug773286.patch

(wrong bug)

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Actually, the smarter solution here is to put this on the new releng cluster.  I'll work on that.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug773057-slavealloc-base_url.patch

catlee, rail-buildduty said you had lots of free time and were looking for something to do ;)

This adds a base_url config to slavealloc that it uses to construct URLs.

This is necessary for this bug, where we want to host slavealloc on https://secure.pub.build.mozilla.org/slavealloc, so it needs to know to produce that URL for references.

I'll have another patch in a bit to also record the LDAP username of people making changes in the UI.  That should help diagnose weirdnesses a little.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 652747 [details] [diff] [review]
bug773057-slavealloc-base_url.patch

checked in and pushed on the existing server.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

OK, on the new web cluster, we now have
 http://slavealloc.pvt.build.mozilla.org/ - allocator
 https://secure.pub.build.mozilla.org/slavealloc - UI + API
   authentication is limited to the RelEngWiki group (c.f. bug 783563)

I need to fix up backups and add monitoring.  Once that's done, we can turn this bug over to releng to change the URLs in runslave.py.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

OK, bug 783665 for monitoring, and the backups are in place.

This new service has the same backend as the old, so it's interchangeable.  Here's what I suggest:

* briefly test out http://slavealloc.pvt.build.mozilla.org/ to make sure it's allocating correctly
* commit puppet & puppetagain changes to point puppety slaves to this host

and, once bug 774354 is complete:

* relengers start using https://secure.pub.build.mozilla.org/slavealloc for the UI
* I make slavealloc.build.mozilla.org an alias for slavealloc.pvt.build.mozilla.org, thereby moving windows slaves over to that service without having to reconfigure them
** add a 301 redirect from http://slavealloc.build.mozilla.org/ to https://secure.pub.build.mozilla.org/slavealloc for ease of awesomebar transitions
* after a week or so, decommission the old slavealloc VM

The priority on this is relatively low, but the effort is also pretty minor.  It gets us away from the soft SPOF of the single-hosted slavealloc system in scl1, and puts the application in a context that SREs and ops are very familiar with (webapp), so overall it's a stability and maintainability win.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Comment 7 is still accurate here (which impressive in itself, 10 months later!).  Bug 774354 has been split out quite a bit in an effort to get traction, but that should be settled next Wednesday.

As I mentioned in comment 7, slavealloc.pvt.b.m.o is working already, if you want to test it out and/or start pointing slaves at it.

Coop, I'm assigning this to you on the basis of your being "motivated" in bug 864455 :)

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

This is a nice, easy change that will let us kill slavealloc01.build.scl1.mozilla.com.

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #9)
> Coop, I'm assigning this to you on the basis of your being "motivated" in
> bug 864455 :)

I *am* motivated, but despite the fact that this would help me do buildduty more effectively, I likely won't have time to tackle this until next week when I'm off of buildduty.

Comment 12

[Chris Cooper [:coop] (he/him)]

Wading back into this after a hiatus...

Reading comment #7, and indeed poking around https://secure.pub.build.mozilla.org/slavealloc (which slave_health is already using) and http://slavealloc.pvt.build.mozilla.org/, makes me guess that the setup work here is already done.

Some questions for you, Dustin:

1) Is the above accurate, i.e. the scope of work here is just to get slaves pointed to http://slavealloc.pvt.build.mozilla.org/ via runslave.py?

2) If I want to update the currently running instance of slavealloc at https://secure.pub.build.mozilla.org/slavealloc, where do I do that?

3) Is there a staging instance of slavealloc in the new location? If not, can I do that trivially in the new location? I'm familiar with the setup script/process from the old slavealloc server.

If #1 is true, I can start pointing slaves at the new target this week, starting with a couple of lion/snow slaves that are already unmanaged that I would need to update manually anyway.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Chris Cooper [:coop] from comment #12)
> 1) Is the above accurate, i.e. the scope of work here is just to get slaves
> pointed to http://slavealloc.pvt.build.mozilla.org/ via runslave.py?

Correct.

> 2) If I want to update the currently running instance of slavealloc at
> https://secure.pub.build.mozilla.org/slavealloc, where do I do that?

That's the same kind of webops push as for trychooser updates.  Basically, you land the changes and then either run a deploy script yourself, or file a webops bug to do so.  For the procedural details I'll point you to webops - I'm not sure what the current recommendation is.

> 3) Is there a staging instance of slavealloc in the new location? If not,
> can I do that trivially in the new location? I'm familiar with the setup
> script/process from the old slavealloc server.

There's not, but now that we've got everything on the new cluster, it's pretty easy to set up staging envs - bug 841345 is the starting point.

> If #1 is true, I can start pointing slaves at the new target this week,
> starting with a couple of lion/snow slaves that are already unmanaged that I
> would need to update manually anyway.

Sounds good!

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

We took care of this in bug 899085.