Closed Bug 1289277 Opened 3 years ago Closed 3 years ago
Reduce amount of buildbot-bridge hosts
58 bytes, text/x-review-board-request
As a work around for race conditions we can try to reduce the amount of BBB hosts from 3 to 2 or even 1.
Review commit: https://reviewboard.mozilla.org/r/67200/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/67200/
Attachment #8774754 - Flags: review?(bugspam.Callek)
The patch won't be deleting anything important, only the supervisor files (and maybe some other configs). This will stop the processes on those masters.
Comment on attachment 8774754 [details] Bug 1289277 - Reduce amount of buildbot-bridge hosts https://reviewboard.mozilla.org/r/67200/#review64026 I'm not really convinced this will help us that much. We have seen some relatively rare race conditions of things running twice, and this can help pinpoint that a bit, while absent other logging. Our underlying issue is lack of good [useful] logging and buildbot RETRY's reusing buildrequestid.
Attachment #8774754 - Flags: review?(bugspam.Callek) → review+
I'm going to land this. We talked with catlee about dupes (bug 1288415) and the idea to add an extra check in handlePending before we add it to the DB, but it requires to be serial, so we have to run a single instance in this case.
Comment on attachment 8774754 [details] Bug 1289277 - Reduce amount of buildbot-bridge hosts remote: https://hg.mozilla.org/build/puppet/rev/f074712cc9bd remote: https://hg.mozilla.org/build/puppet/rev/8d7feedb3722
Attachment #8774754 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
It looks like 1 instance is not enough to process all incoming buildbot events. We had a lot of tasks cancelled by the reflector because of this.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Let's wontfix this...
Status: REOPENED → RESOLVED
Closed: 3 years ago → 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.