936460 - Run selfserve-agent on multiple masters

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

To cancel builds selfserve-agent uses buildbot web interface and this operation sometimes takes a lot of time what blocks other selfserve requests.

To improve this situation we can deploy and run selfserve agent on multiple masters. I tested 2 agents running in parallel on bm36 and bm81 and they behave as expected.

TODO:

* requires rabbit2 netflows (see bug 934505)
* make sure that the masters can talk to each other via buildbot http interface (to cancel builds)
* One liner per master should enable it https://hg.mozilla.org/build/puppet/rev/5acc6e2544b9#l2.12

Comment 1

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #0)
> * requires rabbit2 netflows (see bug 934505)
Nope, already done.

* file bugs to add selfserve-agent nagios checks

Comment 2

[Rail Aliiev [:rail]]

Attachment: [puppet] add bm61 (use1) and bm64 (usw2)

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 829322 [details] [diff] [review]
[puppet] add bm61 (use1) and bm64 (usw2)

https://hg.mozilla.org/build/puppet/rev/68cdbe4916e6

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Do we avoid duplicated actions because each instance removes a message from the queue prior to acting on it ?

Comment 5

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #4)
> Do we avoid duplicated actions because each instance removes a message from
> the queue prior to acting on it ?

Yes. I watched multiple masters last week and didn't see any dupes.

Comment 6

[Rail Aliiev [:rail]]

FTR, I'd like to add 3 more (one per colo) agents sometime next week.

Comment 7

[Rail Aliiev [:rail]]

Attachment: Add 2 more masters

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8338839 [details] [diff] [review]
Add 2 more masters

remote:   https://hg.mozilla.org/build/puppet/rev/28db31fe2760
remote:   https://hg.mozilla.org/build/puppet/rev/84585ced30ca