Closed Bug 1289277 Opened 3 years ago Closed 3 years ago

Reduce amount of buildbot-bridge hosts

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: rail, Assigned: rail)

References

Details

(Whiteboard: [bbb])

Attachments

(1 file)

As a work around for race conditions we can try to reduce the amount of BBB hosts from 3 to 2 or even 1.
The patch won't be deleting anything important, only the supervisor files (and maybe some other configs). This will stop the processes on those masters.
Comment on attachment 8774754 [details]
Bug 1289277 - Reduce amount of buildbot-bridge hosts

https://reviewboard.mozilla.org/r/67200/#review64026

I'm not really convinced this will help us that much. We have seen some relatively rare race conditions of things running twice, and this can help pinpoint that a bit, while absent other logging.

Our underlying issue is lack of good [useful] logging and buildbot RETRY's reusing buildrequestid.
Attachment #8774754 - Flags: review?(bugspam.Callek) → review+
Whiteboard: [bbb]
I'm going to land this.

We talked with catlee about dupes (bug 1288415) and the idea to add an extra check in handlePending before we add it to the DB, but it requires to be serial, so we have to run a single instance in this case.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Blocks: 1296409
It looks like 1 instance is not enough to process all incoming buildbot events. We had a lot of tasks cancelled by the reflector because of this.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Let's wontfix this...
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.