1289277 - Reduce amount of buildbot-bridge hosts

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

As a work around for race conditions we can try to reduce the amount of BBB hosts from 3 to 2 or even 1.

Comment 1

[Rail Aliiev [:rail]]

Attachment: Bug 1289277 - Reduce amount of buildbot-bridge hosts

Review commit: https://reviewboard.mozilla.org/r/67200/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67200/

Comment 2

[Rail Aliiev [:rail]]

The patch won't be deleting anything important, only the supervisor files (and maybe some other configs). This will stop the processes on those masters.

Comment 3

[Justin Wood (:Callek)]

Comment on attachment 8774754 [details]
Bug 1289277 - Reduce amount of buildbot-bridge hosts

https://reviewboard.mozilla.org/r/67200/#review64026

I'm not really convinced this will help us that much. We have seen some relatively rare race conditions of things running twice, and this can help pinpoint that a bit, while absent other logging.

Our underlying issue is lack of good [useful] logging and buildbot RETRY's reusing buildrequestid.

Comment 4

[Rail Aliiev [:rail]]

I'm going to land this.

We talked with catlee about dupes (bug 1288415) and the idea to add an extra check in handlePending before we add it to the DB, but it requires to be serial, so we have to run a single instance in this case.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8774754 [details]
Bug 1289277 - Reduce amount of buildbot-bridge hosts

remote:   https://hg.mozilla.org/build/puppet/rev/f074712cc9bd
remote:   https://hg.mozilla.org/build/puppet/rev/8d7feedb3722

Comment 6

[Rail Aliiev [:rail]]

It looks like 1 instance is not enough to process all incoming buildbot events. We had a lot of tasks cancelled by the reflector because of this.

Comment 7

[Rail Aliiev [:rail]]

Let's wontfix this...