1135818 - Port slavealloc API, UI to relengapi

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

There are three parts to https://wiki.mozilla.org/ReleaseEngineering/Applications/Slavealloc

 * the "allocator" that each of the buildslaves use to get their configuration on startup
 * the API, which allows updating and changing records in the DB
 * the UI, which uses the API on the backend

Currently the API is completely unauthenticated -- if you have access, you can change whatever you want.  That's not great, because it means we can't really trust the data in the database.

The UI is a total piece of junk.  I can say that because I wrote it!  With the number of slaves we have, it hangs the browser when you load it.  Boo.

So the idea here is to rewrite just the API and UI bits using RelengAPI.  The allocator would remain as-is (for now.. that's another project).  The API is really basic: create/read/update/delete slaves and masters.

Comment 1

[Justin Wood (:Callek)]

To say, alongside this, is that the slavealloc changes that take place, have long wanted some form of history and *who changed them*

E.g. if I disable a slave, or mark a comment saying "on loan" would be great to be able to see who did that.

This need is not in current implementation so doesn't block this, but is good to think about while designing its new home.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Let's keep it simple to start, but an audit trail will be a good follow-on project.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: slavealloc.dump

Here's a dump of the current slavealloc DB, with slave passwords scrubbed out.  Hopefully this will just load right into a sqlite database, but it was exported from MySQL so there may be issues.  At worst, it should give you an idea what the data looks like so that you can create your own test data.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Tanuj, how's this going?

Comment 5

[tanujsrivastava22]

I was busy with my University Assessments. Gearing up from today. Will report you soon.

Comment 6

[Pete Moore [:pmoore][:pete]]

(In reply to Dustin J. Mitchell [:dustin] from comment #0)

> So the idea here is to rewrite just the API and UI bits using RelengAPI. 
> The allocator would remain as-is (for now.. that's another project).  The
> API is really basic: create/read/update/delete slaves and masters.

Do we have a definition of this API, i.e. the exact http methods and urls to use, payloads etc?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

It's defined in the code:
  https://github.com/mozilla/build-tools/blob/master/lib/python/slavealloc/daemon/http/api.py
That doesn't specify the payloads, but they're flat JSON dicts so running a copy of the daemon in a test environment should make it easy to find out.

Comment 8

[Pete Moore [:pmoore][:pete]]

I'm wondering if maybe slavealloc becomes less important with a move to put workers in the cloud, and TaskCluster not tracking workers. In other words, maybe we don't need to have a slave database in the "new world". This may still apply for legacy, of course.

In light of this though, should we maybe lower the priority on this one?

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to Pete Moore [:pmoore][:pete] from comment #8)
> I'm wondering if maybe slavealloc becomes less important with a move to put
> workers in the cloud, and TaskCluster not tracking workers. In other words,
> maybe we don't need to have a slave database in the "new world". This may
> still apply for legacy, of course.

Buildbot will still be with us for many months.

Even in the taskcluster world we will need tools for managing hardware machines that Mozilla owns & operates. Slavealloc is not that tool, but a relengapi version thereof that can also accept health data from sources like runner certainly is.

> In light of this though, should we maybe lower the priority on this one?

In effect, we've already done that by soliciting help from a contributor who doesn't work for us. We're happy to manage this and assist, but it's a good opportunity for Tanuj to learn his way around.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Callek, this is another good place to see that merging relengapi blueprints makes sense -- a "buildslaves" table will be useful for more than just this blueprint.