production-ify allthethings.json

RESOLVED WORKSFORME

Status

Release Engineering
General Automation
RESOLVED WORKSFORME
3 years ago
9 months ago

People

(Reporter: catlee, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

3 years ago
We need a real home for the tools that generate allthethings.json to live if we're going to be relying on it in production. It also needs proper error reporting, nagios checks, etc.

Comment 1

3 years ago
I created a wiki page (until this is done) with minimal information I know about:
https://wiki.mozilla.org/ReleaseEngineering/How_To/allthethings.json
Step 1: move code into buildbot-configs repo
Created attachment 8622641 [details] [diff] [review]
bug1131787.patch
Attachment #8622641 - Flags: review?(catlee)
Assignee: nobody → sdeckelmann
First patch moves files into buildbotcustom repo.
(Reporter)

Comment 5

3 years ago
Comment on attachment 8622641 [details] [diff] [review]
bug1131787.patch

Review of attachment 8622641 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm.

does make_allthethings.sh need to be part of this, or should that go in the deployment code? it has fairly specific paths in it.
Attachment #8622641 - Flags: review?(catlee) → review+
Created attachment 8622688 [details] [diff] [review]
bug1131787-puppet.patch

Moved make_allthethings.sh into puppet config and cron under 'buildduty' and a few minor cosmetic updates to scripts. (update to other patch coming in a moment)
Attachment #8622688 - Flags: review?(catlee)
Created attachment 8622689 [details] [diff] [review]
bug1131787-buildbotcustom.patch

r+ from catlee - moved one file into puppet.
Attachment #8622641 - Attachment is obsolete: true
Comment on attachment 8622688 [details] [diff] [review]
bug1131787-puppet.patch

Review of attachment 8622688 [details] [diff] [review]:
-----------------------------------------------------------------

Adding in the comments from irc. We want some failsafes around things like simultaneous run from cron (use lockfile), cleaning up lockfiles/tempp file when interrupted (use trap), making sure we put timeouts in the hg commands so they don't hang forever, and sending the output to syslog instead of cronmail (use logger -t).

I mentioned that looking at the crontask class might be a good fit (most of the aws cron jobs use this, see most of the lines above that one in the cron.pp file) but maybe not everything lines up there.
(Reporter)

Updated

2 years ago
Attachment #8622688 - Flags: review?(catlee) → review+
I was asked to use 'hgtool.py' for hg timeouts. working on it.

Comment 10

2 years ago
Hi Selena, are you still working on this?
Flags: needinfo?(sdeckelmann)
(In reply to Armen Zambrano Gasparnian [:armenzg] from comment #10)
> Hi Selena, are you still working on this?

Ah - I was just thinking about this yesterday. I can poke at it tomorrow and let you know whether or not I can get it into production.
Oops, pushing to next week. Sorry!

Comment 13

2 years ago
There is a script to set up the venv:
http://hg.mozilla.org/build/braindump/file/default/community/setup_buildbot_environment.sh

I'm using on cruncher a hybrid of my script with catlee's:
https://hg.mozilla.org/build/braindump/file/default/community/generate_allthethings_json.sh#l37

We're generating the same file side-by-side with catlee's script:
https://secure.pub.build.mozilla.org/builddata/reports/allthethings.new.json

All of this was explained in here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1210390#c11
(Reporter)

Updated

2 years ago
Duplicate of this bug: 1233855

Updated

2 years ago
Flags: needinfo?(sdeckelmann)
Assignee: sdeckelmann → nobody
A lot of this is in puppet https://dxr.mozilla.org/build-central/source/puppet/modules/cruncher/manifests/allthethings.pp via bug 1261974. Just missing some monitoring ?
We have nagios checks on builddata.pub.build.mozilla.org on /buildjson/builds-4hr.js.gz and friends, so that's a natural place to add a new check for /reports/allthethings.json and
(Reporter)

Updated

9 months ago
Status: NEW → RESOLVED
Last Resolved: 9 months ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.