681834 - Insert finished jobs in the statusdb more frequently

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

On buildbot masters, we insert newly finished builds into the statusdb every 10 minutes. Turns out that's the long pole in getting the data to the buildbot-based tbpl, so we need to see if we can shorten the interval up without spending all our time scanning the disk instead of doing actual work.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Modify new master installs

Looking at the update.log's that cron leaves behind, there are very short runs of 2-4 seconds where no builds are added to the db, or up to 120 seconds if some are. There's quite a bit of variation in the time to insert - adding 15 builds can take as little as 20s or as much as 110s on buildbot-master11. The ganglia graph of CPU usage shows we're not CPU limited, so nice'ing the process isn't relevant, and the variation is probably down to the db server.

I think we can safely turn the cron down to every 2 minutes, because there is a lock guarding starting more instances. If we run into contention issues due to db locking we can revisit. This review is for signoff on modifying the existing master setups too.

Comment 2

[Phil Ringnalda (:philor)]

Still planning on doing this, or is PulseStatus going to save us instead?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 555680 [details] [diff] [review]
[buildbot-configs] Modify new master installs

http://hg.mozilla.org/build/buildbot-configs/rev/a8e417e2c927

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I've updated 
  buildbot-master08:/etc/cron.d/bm08-build1
  buildbot-master11:/etc/cron.d/bm11-tests1
to confirm this will work OK in prod.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

The other masters based have been done too, based on the contents of production-masters.json, enabled or not (bm04, bm06, bm07, bm09, bm12, bm13, bm14, bm15, bm16, test-master01, bm3).

In the case of the two schedulers on bm10 I disabled the cronjob, which had been complaining about a missing config file anyway. bm3 needed the cron to delete stale lockfiles.