buildbot database usage needs to split writes and reads between master and slave

RESOLVED INVALID

Status

Release Engineering
General
RESOLVED INVALID
6 years ago
5 years ago

People

(Reporter: mpressman, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
tm-b01-master01 is the master where writes should go to
and 
tm-b01-slave01 is the slave where reads should go to

This will facilitate better performance and redundancy.
Does buildbot have any support for this already or upstream ?
(In reply to Nick Thomas [:nthomas] from comment #1)
> Does buildbot have any support for this already or upstream ?

our version certainly doesn't

also, we're already using the slave for all our reporting
Upstream has more efficient use of the database, but doesn't have support for splitting out ro and rw access.  The new API is much more consistent in its approach, though, so this could be added fairly easily (two engines, and an option to the pool methods).

As BB maintainer, I'm curious what the read/write ratio is for the 'buildbot_schedulers' db.  Matt, can you find out?

The 'buildbot' db stores build status, and is not part of the upstream code, so I'm less familiar.  AIUI, it is already split into writers and readers -- readers are the reporting catlee's describing in comment 2, and the writers are whatever code slurps up the status pickles and blasts them into the DB.  It'd be good to have this assertion verified, too.

Finally, to clarify comment 0, this particular split doesn't buy us much redundancy, since it's the writes that are critical for this application -- Buildbot can't build anything if it can't write the build requests, etc. to the database.

Still, better performance is important!
from irc: 

mpressman, cshields feel that this would have reduced the number of things which were involved in Monday's 6h tree closure. There were other (more invasive?) things going wrong, so I feel it unlikely this would have been enough for us to avoid a treeclosure. See details in bug#702393 as we unwind what happened.
See Also: → bug 702393
As :dustin pointed out in comment 3, this won't actually result in any resiliency for buildbot, since it's a write-heavy application. Marking INVALID and filed bug 707203 for the new cluster.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → INVALID
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.