660146 - change clientproxy to use runslave.py's routines to retrieve buildbot.tac

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Mike Taylor [:bear]]

doing this will enable tegra's to be managed in a partial way by slavealloc

see if runslave.py can be imported and use it's routine to fetch the tac file

Comment 1

[Mike Taylor [:bear]]

marking this as WONTFIX because we are moving to the new foopy-less world where clientproxy is no longer required

Comment 2

[Armen [:armenzg]]

I believe this is important to do even if we will eventually (I don't see it happening soon) move to foopy-less tegras.

The reason is because it is a lot of work to move tegras across masters (see bug 734393) and there is a high utilization of buildbot-master19 and buildbot-master20 [1]

We have there spikes of CPU wio and I have seen clean up steps to time out because the foopy times out removing files.

If there is a bug for tracking switching to the foopy-less setup we can determine if we want to wait or not.

[1] http://ganglia3.build.mtv1.mozilla.com/ganglia/?gw=fwd&gs=RelEng%20scl1%40http%3A%2F%2Fganglia1.build.scl1.mozilla.com%2Fganglia%2F

Comment 3

[Aki Sasaki (not active)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #2)
> I believe this is important to do even if we will eventually (I don't see it
> happening soon) move to foopy-less tegras.

Foopy-less tegras, as I understand it, won't be talking directly to buildbot at all.
Instead, a build- or test-slave will be running buildbot, and calling a mozharness script that will talk to a tegra from a pool.

If you've got a different idea or understanding about getting rid of the foopies, I'd be interested in hearing about it.

Comment 4

[Armen [:armenzg]]

That is pretty neat. This means we would have more load on the test slaves.
Is there a bug for that? Perhaps using the rev4 testing machines (since there are many of them) might be right OS to target to do so.

Back to this bug, I believe it is valid to do this after what I mention in comment 2; do you think so?

Comment 5

[Aki Sasaki (not active)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #2)
> I believe this is important to do even if we will eventually (I don't see it
> happening soon) move to foopy-less tegras.

Oops, I misread this to say "important to do *to* eventually move to foopy-less tegras".  Never mind me :)

Comment 6

[Justin Wood (:Callek)]

This could have some large benefits, but I'm not sure if the benefits on a per-tegra basis, vs benefits on a per-foopy basis, so CC'ing kim incase she has free time and thoughts for a slightly-more-complex mobile bug.

bear at least, is surely not still working on this

Comment 7

[Chris Cooper [:coop] (he/him)]

This bug is close enough to the one I wanted to file that I thing I'll just comment here.

We should start using slavealloc to provide buildbot.tac files, but I mean *ALL THE WAY* including enabling/disabling of tegras. This means several things:

* we'll need to make runslave.py importable (if it is not currently) so we're not duplicating code
* clientproxy.py needs to explicitly check that the buildbot.tac is even present before blindly passing it as an arg to twistd: https://hg.mozilla.org/build/tools/file/8a2c29db72ad/sut_tools/clientproxy.py#l383
* even if we're not executing the code in the .tac file directly, clientproxy.py should still be able to grep the file for the "SLAVE DISABLED; NOT STARTING" message and honour it.
* we need to log all these new actions so they are easily identifiable from the dashboard.
* to be mindful of the current SOP for doing software updates on foopys, we'll also need to create a script that can be run from a given foopy (or any machine connected to the build network, really) that can disable/enable all tegras for a given foopy in slavealloc (setting a corresponding Note for each in the db, of course)

Comment 8

[Mike Taylor [:bear]]

resolving as INCOMPLETE because these bugs are most likely not even being worked on or known about anymore

Comment 9

[Justin Wood (:Callek)]

reopenning, c.f. c#7

Comment 10

[[e-mail] (:bhearsum)]

Is this still an issue in the new world where tegra/pandas use slavealloc?

Comment 11

[Justin Wood (:Callek)]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
> Is this still an issue in the new world where tegra/pandas use slavealloc?

It's no longer an issue because tegras/pandas are no longer using clientproxy