The default bug view has changed. See this FAQ.

Add nagios check for age of buildbot process

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: nthomas, Assigned: dgarvey)

Tracking

(Blocks: 1 bug)

Details

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
In bug 1049227 we had some masters where buildbot had been running for 3 months, and the memory consumption had gradually increased to the point that buildbot was really slow and using almost all the RAM. 

Please add a process age check to nagios that alerts after one month. Looks like
 /usr/lib64/nagios/plugins/check_procs -m ELAPSED -w 2678400 -C buildbot
would do the trick.

Puppet patch coming up.
(Reporter)

Comment 1

3 years ago
Created attachment 8476275 [details] [diff] [review]
[puppet] Add check on masters

Warn after 1 month. OK with you coop ?
Attachment #8476275 - Flags: review?(coop)
(Reporter)

Updated

3 years ago
See Also: → bug 1049227
Assignee: relops → server-ops
Component: RelOps → Server Operations
Product: Infrastructure & Operations → mozilla.org
QA Contact: arich → shyam

Updated

3 years ago
Attachment #8476275 - Flags: review?(coop) → review+
(Reporter)

Comment 2

3 years ago
Comment on attachment 8476275 [details] [diff] [review]
[puppet] Add check on masters

Landed:
https://hg.mozilla.org/build/puppet/rev/14d2f770dc65
https://hg.mozilla.org/build/puppet/rev/2acea504ac23

Server ops, please add a check that uses command check_buildbot_age on all members of the buildbot master host group (sp?). Doesn't need any parameters passed in.
(Assignee)

Comment 3

3 years ago
nthomas,

Are these the hostgroups in question?

 'dev-buildbot-masters',
 'scl3-production-buildbot-masters',
 'use1-production-buildbot-masters',
 'usw2-production-buildbot-masters'
(Reporter)

Comment 4

3 years ago
(In reply to dgarvey :dgarvey from comment #3)
>  'scl3-production-buildbot-masters',
>  'use1-production-buildbot-masters',
>  'usw2-production-buildbot-masters'

These ones please.
(Assignee)

Comment 5

3 years ago
I have access to the hosts this time;)

[dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ cat /etc/nagios/nrpe.d/buildbot.cfg 
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

command[check_buildbot]=/usr/lib64/nagios/plugins/check_procs -c 2:2 -C buildbot
command[check_buildbot_age]=/usr/lib64/nagios/plugins/check_procs -m ELAPSED -w 2678400 -C buildbot
[dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ /usr/lib64/nagios/plugins/check_procs -m ELAPSED -w 2678400 -C buildbot
ELAPSED WARNING: 2 warn out of 2 processes with command name 'buildbot'
[dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ echo $?
1
[dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ ps -efww | grep nrpe
dgarvey  20232 16987  0 22:06 pts/0    00:00:00 grep nrpe
nagios   26450     1  0 14:33 ?        00:00:01 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
[dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ cat /etc/nagios/nrpe.cfg
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

####  This file under configuration management control
####  DO NOT EDIT MANUALLY

pid_file=/var/run/nrpe.pid
server_port=5666
nrpe_user=nagios
nrpe_group=nagios
allowed_hosts=127.0.0.1,10.26.75.30
dont_blame_nrpe=1
include_dir=/etc/nagios/nrpe.d
command_timeout=60

# basic, common checks
command[check_load]=/usr/lib64/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_users]=/usr/lib64/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
command[check_inodes]=/usr/lib64/nagios/plugins/check_disk -W $ARG1$ -C $ARG2$ -p $ARG3$
command[check_procs]=/usr/lib64/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
command[check_file_age]=/usr/lib64/nagios/plugins/check_file_age $ARG1$

[dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ dig -x 10.26.75.30

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6_2.2 <<>> -x 10.26.75.30
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59289
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;30.75.26.10.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
30.75.26.10.in-addr.arpa. 3600	IN	PTR	nagios1.private.releng.scl3.mozilla.com.

;; AUTHORITY SECTION:
75.26.10.in-addr.arpa.	300	IN	NS	ns2.mozilla.org.
75.26.10.in-addr.arpa.	300	IN	NS	ns1.mozilla.org.

;; ADDITIONAL SECTION:
ns1.mozilla.org.	60	IN	A	63.245.215.5
ns2.mozilla.org.	60	IN	A	63.245.218.7

;; Query time: 13 msec
;; SERVER: 10.26.75.40#53(10.26.75.40)
;; WHEN: Thu Aug 21 22:07:21 2014
;; MSG SIZE  rcvd: 174

[dgarvey@buildbot-master81.srv.releng.scl3.mozilla.com ~]$ 


dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$ svn commit -m "updated for bug 1056348"
X11 forwarding request failed on channel 0
Sending        manifests/mozilla/checkcommands.pp
Sending        manifests/releng/services.pp
Transmitting file data ..
Committed revision 92348.
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$
Getting alerts in #buildduty channel of IRC. Is this related?

nagios-releng
08:41:58 Thu 23:41:58 PDT [4565] buildbot-master102.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:41:59 Thu 23:41:58 PDT [4566] buildbot-master100.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:41:59 Thu 23:41:58 PDT [4567] buildbot-master106.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:41:59 Thu 23:41:58 PDT [4568] buildbot-master104.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:02 Thu 23:41:58 PDT [4569] buildbot-master108.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:05 Thu 23:41:58 PDT [4570] buildbot-master112.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:08 Thu 23:41:58 PDT [4571] buildbot-master05.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:11 Thu 23:41:59 PDT [4572] buildbot-master52.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:14 Thu 23:41:59 PDT [4573] buildbot-master67.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:17 Thu 23:41:59 PDT [4574] buildbot-master01.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:20 Thu 23:41:59 PDT [4575] buildbot-master03.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:23 Thu 23:41:59 PDT [4576] buildbot-master84.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:26 Thu 23:41:59 PDT [4577] buildbot-master82.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:29 Thu 23:41:59 PDT [4578] buildbot-master79.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:32 Thu 23:41:59 PDT [4579] buildbot-master73.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:35 Thu 23:41:59 PDT [4580] buildbot-master75.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:38 Thu 23:41:59 PDT [4581] buildbot-master77.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:41 Thu 23:41:59 PDT [4582] buildbot-master71.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:44 Thu 23:41:59 PDT [4583] buildbot-master88.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:47 Thu 23:41:59 PDT [4584] buildbot-master99.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:50 Thu 23:42:00 PDT [4585] buildbot-master86.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:42:53 Thu 23:42:00 PDT [4586] buildbot-master91.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:44:49 Thu 23:44:48 PDT [4587] cruncher.srv.releng.scl3.mozilla.com:Pending builds is WARNING: WARNING Pending Builds: 4417 (http://m.mozilla.org/Pending+builds)
08:46:09 Thu 23:46:09 PDT [4588] buildbot-master101.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:09 Thu 23:46:09 PDT [4589] buildbot-master103.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:09 Thu 23:46:09 PDT [4590] buildbot-master06.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:09 Thu 23:46:09 PDT [4591] buildbot-master02.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:12 Thu 23:46:09 PDT [4592] buildbot-master04.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:15 Thu 23:46:09 PDT [4593] buildbot-master105.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:18 Thu 23:46:09 PDT [4594] buildbot-master107.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:21 Thu 23:46:09 PDT [4595] buildbot-master117.bb.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:24 Thu 23:46:19 PDT [4596] buildbot-master83.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:27 Thu 23:46:19 PDT [4597] buildbot-master85.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:30 Thu 23:46:19 PDT [4598] buildbot-master89.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:33 Thu 23:46:19 PDT [4599] buildbot-master87.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:36 Thu 23:46:19 PDT [4600] buildbot-master81.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 2 warn out of 2 processes with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:39 Thu 23:46:19 PDT [4601] buildbot-master72.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:42 Thu 23:46:19 PDT [4602] buildbot-master53.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:45 Thu 23:46:19 PDT [4603] buildbot-master68.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:48 Thu 23:46:19 PDT [4604] buildbot-master74.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:51 Thu 23:46:19 PDT [4605] buildbot-master78.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:54 Thu 23:46:20 PDT [4606] buildbot-master51.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:46:57 Thu 23:46:20 PDT [4607] buildbot-master70.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:47:00 Thu 23:46:20 PDT [4608] buildbot-master94.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
08:47:03 Thu 23:46:20 PDT [4609] buildbot-master76.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
nagios-releng
09:11:23 Fri 00:11:23 PDT [4610] cruncher.srv.releng.scl3.mozilla.com:Pending builds is WARNING: WARNING Pending Builds: 4318 (http://m.mozilla.org/Pending+builds)
09:15:04 Fri 00:15:03 PDT [4611] buildbot-master68.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:15:04 Fri 00:15:03 PDT [4612] buildbot-master67.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:15:04 Fri 00:15:03 PDT [4613] buildbot-master70.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:15:04 Fri 00:15:03 PDT [4614] buildbot-master72.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:15:14 Fri 00:15:13 PDT [4615] buildbot-master81.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 2 warn out of 2 processes with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
nigelb|sheriffduty
09:17:38 pmoore: ^^ might want to look at tht :)
nigelb|sheriffduty
09:17:40 *that
nagios-releng
09:18:53 Fri 00:18:53 PDT [4617] buildbot-master01.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:18:53 Fri 00:18:53 PDT [4618] buildbot-master02.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:18:53 Fri 00:18:53 PDT [4619] buildbot-master04.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:18:54 Fri 00:18:54 PDT [4620] buildbot-master03.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:18:56 Fri 00:18:54 PDT [4621] buildbot-master05.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:18:59 Fri 00:18:54 PDT [4622] buildbot-master06.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:03 Fri 00:19:03 PDT [4623] buildbot-master100.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:05 Fri 00:19:03 PDT [4624] buildbot-master101.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:08 Fri 00:19:08 PDT [4625] buildbot-master102.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:11 Fri 00:19:09 PDT [4626] buildbot-master103.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:14 Fri 00:19:09 PDT [4627] buildbot-master104.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:17 Fri 00:19:09 PDT [4628] buildbot-master105.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:20 Fri 00:19:09 PDT [4629] buildbot-master107.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:23 Fri 00:19:09 PDT [4630] buildbot-master106.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:26 Fri 00:19:09 PDT [4631] buildbot-master108.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:29 Fri 00:19:13 PDT [4632] buildbot-master112.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:32 Fri 00:19:14 PDT [4633] buildbot-master117.bb.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:35 Fri 00:19:14 PDT [4634] buildbot-master51.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:38 Fri 00:19:14 PDT [4635] buildbot-master52.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:41 Fri 00:19:14 PDT [4636] buildbot-master53.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:44 Fri 00:19:23 PDT [4637] buildbot-master71.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:47 Fri 00:19:24 PDT [4638] buildbot-master73.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:50 Fri 00:19:24 PDT [4639] buildbot-master74.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:53 Fri 00:19:24 PDT [4640] buildbot-master75.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:56 Fri 00:19:24 PDT [4641] buildbot-master76.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:19:59 Fri 00:19:33 PDT [4642] buildbot-master78.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:02 Fri 00:19:34 PDT [4643] buildbot-master79.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:05 Fri 00:19:34 PDT [4644] buildbot-master77.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:08 Fri 00:19:34 PDT [4645] buildbot-master82.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:11 Fri 00:19:34 PDT [4646] buildbot-master83.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:14 Fri 00:19:34 PDT [4647] buildbot-master84.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:17 Fri 00:19:34 PDT [4648] buildbot-master85.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:20 Fri 00:19:34 PDT [4649] buildbot-master86.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:23 Fri 00:19:34 PDT [4650] buildbot-master87.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:26 Fri 00:19:34 PDT [4651] buildbot-master88.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:29 Fri 00:19:44 PDT [4652] buildbot-master89.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:32 Fri 00:19:44 PDT [4653] buildbot-master91.srv.releng.usw2.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:35 Fri 00:19:44 PDT [4654] buildbot-master99.srv.releng.scl3.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
09:20:38 Fri 00:19:44 PDT [4655] buildbot-master94.srv.releng.use1.mozilla.com:buildbot masters age is WARNING: ELAPSED WARNING: 1 warn out of 1 process with command name buildbot (http://m.mozilla.org/buildbot+masters+age)
(Reporter)

Comment 7

3 years ago
Yup, this is a new regime of catching masters before they head off into the weeds by running for long times. I'll downtime the checks until my Monday and do a rolling graceful restart then.
(Assignee)

Comment 8

3 years ago
I think this is fixed nthomas? closing.
Assignee: server-ops → dgarvey
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Blocks: 1057888
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.