1131787 - production-ify allthethings.json

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

We need a real home for the tools that generate allthethings.json to live if we're going to be relying on it in production. It also needs proper error reporting, nagios checks, etc.

Comment 1

[Armen [:armenzg]]

I created a wiki page (until this is done) with minimal information I know about:
https://wiki.mozilla.org/ReleaseEngineering/How_To/allthethings.json

Comment 2

[Selena Deckelmann :selenamarie :selena]

Step 1: move code into buildbot-configs repo

Comment 3

[Selena Deckelmann :selenamarie :selena]

Attachment: bug1131787.patch

Comment 4

[Selena Deckelmann :selenamarie :selena]

First patch moves files into buildbotcustom repo.

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 8622641 [details] [diff] [review]
bug1131787.patch

Review of attachment 8622641 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm.

does make_allthethings.sh need to be part of this, or should that go in the deployment code? it has fairly specific paths in it.

Comment 6

[Selena Deckelmann :selenamarie :selena]

Attachment: bug1131787-puppet.patch

Moved make_allthethings.sh into puppet config and cron under 'buildduty' and a few minor cosmetic updates to scripts. (update to other patch coming in a moment)

Comment 7

[Selena Deckelmann :selenamarie :selena]

Attachment: bug1131787-buildbotcustom.patch

r+ from catlee - moved one file into puppet.

Comment 8

[Amy Rich [:arr] [:arich]]

Comment on attachment 8622688 [details] [diff] [review]
bug1131787-puppet.patch

Review of attachment 8622688 [details] [diff] [review]:
-----------------------------------------------------------------

Adding in the comments from irc. We want some failsafes around things like simultaneous run from cron (use lockfile), cleaning up lockfiles/tempp file when interrupted (use trap), making sure we put timeouts in the hg commands so they don't hang forever, and sending the output to syslog instead of cronmail (use logger -t).

I mentioned that looking at the crontask class might be a good fit (most of the aws cron jobs use this, see most of the lines above that one in the cron.pp file) but maybe not everything lines up there.

Comment 9

[Selena Deckelmann :selenamarie :selena]

I was asked to use 'hgtool.py' for hg timeouts. working on it.

Comment 10

[Armen [:armenzg]]

Hi Selena, are you still working on this?

Comment 11

[Selena Deckelmann :selenamarie :selena]

(In reply to Armen Zambrano Gasparnian [:armenzg] from comment #10)
> Hi Selena, are you still working on this?

Ah - I was just thinking about this yesterday. I can poke at it tomorrow and let you know whether or not I can get it into production.

Comment 12

[Selena Deckelmann :selenamarie :selena]

Oops, pushing to next week. Sorry!

Comment 13

[Armen [:armenzg]]

There is a script to set up the venv:
http://hg.mozilla.org/build/braindump/file/default/community/setup_buildbot_environment.sh

I'm using on cruncher a hybrid of my script with catlee's:
https://hg.mozilla.org/build/braindump/file/default/community/generate_allthethings_json.sh#l37

We're generating the same file side-by-side with catlee's script:
https://secure.pub.build.mozilla.org/builddata/reports/allthethings.new.json

All of this was explained in here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1210390#c11

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

A lot of this is in puppet https://dxr.mozilla.org/build-central/source/puppet/modules/cruncher/manifests/allthethings.pp via bug 1261974. Just missing some monitoring ?

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

We have nagios checks on builddata.pub.build.mozilla.org on /buildjson/builds-4hr.js.gz and friends, so that's a natural place to add a new check for /reports/allthethings.json and