Closed Bug 787185 Opened 12 years ago Closed 12 years ago

set up nagging for db servers that have a new my.cnf but haven't restarted

Categories

(Data & BI Services Team :: DB: MySQL, task)

Product:
Data & BI Services Team
Component:
DB: MySQL
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

(
URL
)

Details

(Whiteboard: [2012q4]) 

It'd be great if this could also catch runtime global variables that are set to something other than their defaults, but not recorded in my.cnf.  We should either get a dbnotices email or, better (because we can downtime it) a nagios alert when a host has been running with a different my.cnf for over, say, 4 hours.
Giving to Dustin. We have not set a goal/timeframe for this (that's OK, but if we want to, we can).
Assignee: server-ops-database → dustin
Depends on: 793320
Well, now that the toolkit is installed:

[root@engagement1.db.phx1 dmitchell]# pt-config-diff /etc/my.cnf h=localhost
1 config difference
Variable                  /etc/my.cnf engagement1.db.phx1.mozilla.com
========================= =========== ===============================
long_query_time           3           2.000000

(manually edited my.cnf; without that, no diffs)

----

So, I'm going to wrap this into a crontask, but only send the results to myself for a while.
Whiteboard: [2012q4]
Nagging should be done by Nagios; machines should have puppet installing percona toolkit, so it might be something to be done with nrpe. rtucker can help figure it out and/or write a script for it.

(if we do that, let's make sure we can release the script to the world. I have no problem putting it on nagiosplugins.org or something...)
The crontask will be easier to work out kinks without paging people.  Turning it into a nagios script is pretty trivial, but I'll be glad to publish it.
Wow, *lots* of diffs.

Variable                  /etc/my.cnf db2.stage.bugs.scl3.mozilla.com
========================= =========== ===============================
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf db2.stage.bugs.scl3.mozilla.com
========================= =========== ===============================
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf stage2.db.scl3.mozilla.com
========================= =========== ==========================
wait_timeout              120         600
Variable                  /etc/my.cnf                               desktop-di
========================= ========================================= ==========
character_set_server      utf8                                      latin1
expire_logs_days          10                                        0
ft_min_word_len           2                                         4
innodb_buffer_pool_size   536870912                                 8388608
innodb_file_per_table     ON                                        OFF
innodb_flush_log_at_tr... 2                                         1
innodb_log_buffer_size    52428800                                  1048576
innodb_log_file_size      314572800                                 5242880
interactive_timeout       600                                       28800
join_buffer_size          8388608                                   131072
key_buffer_size           536870912                                 8384512
log_bin                   /var/lib/mysql/desktop-disk-encrypt-bin   OFF
log_slave_updates         ON                                        OFF
long_query_time           2                                         10.000000
max_allowed_packet        33554432                                  1048576
max_connections           1200                                      151
net_buffer_length         32768                                     16384
preload_buffer_size       2097152                                   32768
query_cache_type          0                                         ON
read_buffer_size          8388608                                   131072
read_rnd_buffer_size      4194304                                   262144
relay_log                 /var/lib/mysql/desktop-disk-encrypt-re...
relay_log_index           /var/lib/mysql/desktop-disk-encrypt-re...
server_id                 22074157                                  0
skip_name_resolve         ON                                        OFF
table_open_cache          3072                                      64
thread_cache_size         500                                       0
tmp_table_size            33554432                                  16777216
wait_timeout              600                                       28800
Variable                  /etc/my.cnf intranet2.db.corp.phx1.mozilla.com
========================= =========== ==================================
expire_logs_days          10          0
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf intranet2.stage.db.corp.phx1.mozilla.com
========================= =========== ========================================
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf intranet1.db.corp.phx1.mozilla.com
========================= =========== ==================================
expire_logs_days          10          0
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf intranet1.stage.db.corp.phx1.mozilla.com
========================= =========== ========================================
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf addons3.db.phx1.mozilla.com
========================= =========== ===========================
binlog_format             MIXED       STATEMENT
Variable                  /etc/my.cnf addons3.versioncheck.db.phx1.mozilla.com
========================= =========== ========================================
innodb_flush_log_at_tr... 2           1
Variable                  /etc/my.cnf addons3.stage.db.phx1.mozilla.com
========================= =========== =================================
auto_increment_increment  2           1
Variable                  /etc/my.cnf dev2.db.scl3.mozilla.com
========================= =========== ========================
ft_min_word_len           2           4
innodb_flush_log_at_tr... 1           0
innodb_flush_method       O_DIRECT    
sync_binlog               1           0
Variable                  /etc/my.cnf builder-addons2.db.scl3.mozilla.com
========================= =========== ===================================
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf addons5.db.phx1.mozilla.com
========================= =========== ===========================
binlog_format             MIXED       STATEMENT
Variable                  /etc/my.cnf tp-bugs01-slave03.phx.mozilla.com
========================= =========== =================================
expire_logs_days          10          2
Variable                  /etc/my.cnf addons4.db.phx1.mozilla.com
========================= =========== ===========================
binlog_format             MIXED       STATEMENT
Variable                  /etc/my.cnf mysql1.metrics.scl3.mozilla.com
========================= =========== ===============================
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
wait_timeout              14401       14400
Variable                  /etc/my.cnf addons1.stage.db.phx1.mozilla.com
========================= =========== =================================
innodb_flush_log_at_tr... 1           2
innodb_flush_method       O_DIRECT    
sync_binlog               1           0
Variable                  /etc/my.cnf node36.seamicro.phx1.mozilla.com
========================= =========== ================================
basedir                   /var/lib    /usr/
Variable                  /etc/my.cnf b1-db1.db.scl3.mozilla.com
========================= =========== ==========================
innodb_flush_log_at_tr... 1           0
innodb_flush_method       O_DIRECT    
key_buffer_size           536870912   2147483648
sync_binlog               1           0
Variable                  /etc/my.cnf buildbot1.db.scl3.mozilla.com
========================= =========== =============================
ft_min_word_len           2           4
Variable                  /etc/my.cnf getpersonas2.db.scl3.mozilla.com
========================= =========== ================================
max_connections           1200        1500
Variable                  /etc/my.cnf builder-addons1.db.scl3.mozilla.com
========================= =========== ===================================
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf engagement2.db.phx1.mozilla.com
========================= =========== ===============================
innodb_flush_log_at_tr... 1           2
innodb_flush_method       O_DIRECT    
sync_binlog               1           0
Variable                  /etc/my.cnf addons7.db.phx1.mozilla.com
========================= =========== ===========================
binlog_format             MIXED       STATEMENT
Variable                  /etc/my.cnf addons6.db.phx1.mozilla.com
========================= =========== ===========================
binlog_format             MIXED       STATEMENT
Variable                  /etc/my.cnf bedrock1.db.scl3.mozilla.com
========================= =========== ============================
expire_logs_days          10          0
ft_min_word_len           2           4
innodb_flush_log_at_tr... 1           0
innodb_flush_method       O_DIRECT    
sync_binlog               1           0
wait_timeout              600         820
Variable                  /etc/my.cnf tp-bugs01-master01.phx.mozilla.com
========================= =========== ==================================
expire_logs_days          10          2
max_connections           4800        1200
Variable                  /etc/my.cnf plugins1.db.phx1.mozilla.com
========================= =========== ============================
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf buildbot2.db.scl3.mozilla.com
========================= =========== =============================
ft_min_word_len           2           4
Variable                  /etc/my.cnf addons2.db.phx1.mozilla.com
========================= =========== ===========================
binlog_format             MIXED       STATEMENT
Variable                  /etc/my.cnf plugins2.db.phx1.mozilla.com
========================= =========== ============================
ft_min_word_len           2           4
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf dev1.db.scl3.mozilla.com
========================= =========== ========================
expire_logs_days          10          0
ft_min_word_len           2           4
innodb_flush_log_at_tr... 1           0
innodb_flush_method       O_DIRECT    
sync_binlog               1           0

pt-config-diff not found (rpm problems, most likely):
 b2-db1.db.scl3.mozilla.com:

Access denied on:
 tp-bugs01-slave01.phx.mozilla.com
 tp-bugs01-slave02.phx.mozilla.com
OK, here are the raw results, more readable in the etherpad:

KEY:
 (ETC) = /etc/my.cnf wins; runtime value updated
 (RUN) = runtime value wins; /etc/my.cnf updated via puppet
 (IGN) = ignorable (just ft_min_word_len)

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
>> db2.stage.bugs.scl3.mozilla.com
> ft_min_word_len           2           4
(ETC) (restarted)
> innodb_flush_log_at_tr... 2           0
(ETC)
>> stage2.db.scl3.mozilla.com
> wait_timeout              120         600
(ETC)
>> desktop-disk-encryption
  all ignored - this is moving to b2 soon
>> intranet2.db.corp.phx1.mozilla.com
> expire_logs_days          10          0
(RUN)
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 2           0
(ETC)
>> intranet1.db.corp.phx1.mozilla.com
> expire_logs_days          10          0
(RUN)
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 2           0
(ETC)
>> intranet2.stage.db.corp.phx1.mozilla.com
> innodb_flush_log_at_tr... 2           0
(ETC)
>> intranet1.stage.db.corp.phx1.mozilla.com
> innodb_flush_log_at_tr... 2           0
(ETC)
>> addons3.versioncheck.db.phx1.mozilla.com
> innodb_flush_log_at_tr... 2           1
(RUN) (was already resolved when I got there)
>> dev1.db.scl3.mozilla.com
> expire_logs_days          10          0
(ETC)
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 1           0
> innodb_flush_method       O_DIRECT    
> sync_binlog               1           0
(just needs a restart)
>> dev2.db.scl3.mozilla.com
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 1           0
> innodb_flush_method       O_DIRECT    
> sync_binlog               1           0
(just needs a restart)
>> builder-addons2.db.scl3.mozilla.com
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 2           0
(ETC)
>> tp-bugs01-master01.phx.mozilla.com
> expire_logs_days          10          2
(RUN)
> max_connections           4800        1200
(ETC)
>> tp-bugs01-slave01.phx.mozilla.com
> expire_logs_days          10          2   
(RUN)
> innodb_buffer_pool_size   23622320128 19327352832
(ETC) - needs restart, but this is the master
> max_connections           4800        1200
(ETC)
>> tp-bugs01-slave02.phx.mozilla.com
> expire_logs_days          10          2   
(RUN)
>> tp-bugs01-slave03.phx.mozilla.com
> expire_logs_days          10          2
(RUN)
>> mysql1.metrics.scl3.mozilla.com
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 2           0
(ETC)
> wait_timeout              14401       14400
(RUN)
>> addons1.stage.db.phx1.mozilla.com
> innodb_flush_log_at_tr... 1           2
> innodb_flush_method       O_DIRECT    
> sync_binlog               1           0
(just needs a restart)
>> addons3.stage.db.phx1.mozilla.com
> auto_increment_increment  2           1
(ETC)
>> node36.seamicro.phx1.mozilla.com
> basedir                   /var/lib    /usr/
(ETC)
>> b1-db1.db.scl3.mozilla.com
> innodb_flush_log_at_tr... 1           0
> innodb_flush_method       O_DIRECT    
> sync_binlog               1           0
(just needs a restart)
> key_buffer_size           536870912   2147483648
(ETC)
>> buildbot1.db.scl3.mozilla.com
> ft_min_word_len           2           4
(IGN)
>> buildbot2.db.scl3.mozilla.com
> ft_min_word_len           2           4
(IGN)
>> getpersonas2.db.scl3.mozilla.com
> max_connections           1200        1500
(RUN)
>> builder-addons1.db.scl3.mozilla.com
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 2           0
(ETC)
>> engagement2.db.phx1.mozilla.com
> innodb_flush_log_at_tr... 1           2
> innodb_flush_method       O_DIRECT    
> sync_binlog               1           0
(just needs a restart)
>> addons7.db.phx1.mozilla.com
> binlog_format             MIXED       STATEMENT
>> addons6.db.phx1.mozilla.com
> binlog_format             MIXED       STATEMENT
>> addons5.db.phx1.mozilla.com
> binlog_format             MIXED       STATEMENT
>> addons4.db.phx1.mozilla.com
> binlog_format             MIXED       STATEMENT
>> addons3.db.phx1.mozilla.com
> binlog_format             MIXED       STATEMENT
>> addons2.db.phx1.mozilla.com
> binlog_format             MIXED       STATEMENT
>> bedrock1.db.scl3.mozilla.com
> expire_logs_days          10          0
(RUN) (updated for bedrock2 as well)
> ft_min_word_len           2           4
(IGN)
> wait_timeout              600         820
(RUN) (updated for bedrock2 as well)
> innodb_flush_log_at_tr... 1           0
> innodb_flush_method       O_DIRECT    
> sync_binlog               1           0
(just needs restart)
>> plugins1.db.phx1.mozilla.com
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 2           0
(ETC)
>> plugins2.db.phx1.mozilla.com
> ft_min_word_len           2           4
(IGN)
> innodb_flush_log_at_tr... 2           0
(ETC)
> pt-config-diff not found
>  b2-db1.db.scl3.mozilla.com:
rpm problems..
I temporarily set the crontask to ignore ft_min_word_len, pending restarting servers with the wrong value.
Open questions (for sheeri):

* all but one of the addons hosts has the wrong binlog_format. Should I just change that in puppet and restart addons1?  Or should they really be using MIXED format?  If the latter, how should I go about converting them?

* All of the hosts with the wrong innodb_flush_method will, I assume need a restart to get the correct settings.  I'm surprised so many are in the wrong state.  Was this changed recently in puppet?

To-Do before we can turn this cronspam loose on dbnotices:

* tp-bugs01-slave01.phx.mozilla.com (awesome name for a master!) has the wrong innodb_buffer_pool_size, so that will need a restart.

* fix RPM problems on b2-db1.db.scl3

To-Do before we can add a nagios check:

* get ft_min_word_len sorted out everywhere (how on earth did this get set wrong in so many places?), probably with some selective slave restarts.
(In reply to Dustin J. Mitchell [:dustin] from comment #8)
> Open questions (for sheeri):
> 
> * all but one of the addons hosts has the wrong binlog_format. Should I just
> change that in puppet and restart addons1?  Or should they really be using
> MIXED format?  If the latter, how should I go about converting them? 

Oh! well, only the master really NEEDS binlog_format. I think at one point they'd gotten rebooted. So long as addons7 has the right binlog format, there's absolutely no harm in changing the rest dynamically.

> * All of the hosts with the wrong innodb_flush_method will, I assume need a
> restart to get the correct settings.  I'm surprised so many are in the wrong
> state.  Was this changed recently in puppet?

Um....not in the last few weeks, that I was aware of. Is there an easy way to see when it was changed?

> To-Do before we can turn this cronspam loose on dbnotices:

> * tp-bugs01-slave01.phx.mozilla.com (awesome name for a master!) has the
> wrong innodb_buffer_pool_size, so that will need a restart.

this is the current master, we're going to switch back to tp-bugs01-master01 as the master when we upgrade to 5.5, which we will do before end of October. And this is why we name our machines 1 and 2 now, not master/slave. :D

it's 22G in /etc/my.cnf vs. 18G in the running config. I'd say we don't need to wait for this to be fixed (e.g. if we put the cron in place, we can ignore the cronspam - if it gets to be a nagios check, we can ack it/downtime it for a few weeks). 


> * fix RPM problems on b2-db1.db.scl3

agreed.

> To-Do before we can add a nagios check:
> 
> * get ft_min_word_len sorted out everywhere (how on earth did this get set
> wrong in so many places?), probably with some selective slave restarts.

default is 4, bugzilla needs it to be 2. it really doesn't matter for anyone other than bugzilla. I think ft_min_word_length in puppet pre-dates me, so it would likely be easier to take out ft_min_word_length and make it a variable, and only have it running for certain machines (at the very least, bugzilla).
OK, so for binlog_format, the options are:
 - set all of them to STATEMENT (which is only a change on addons1)
 - set all of them to MIXED (which mpressman's experimentation suggests, but doesn't prove, is safe)

The docs here are confusing.  They say not to change this value while replication is running, because the value on masters and slaves will get out of sync.  But I don't see how the value matters on the slaves.  There's also a risk of switching modes when using temporary tables.  From my read of the docs, all of these risks are between ROW and STATEMENT; MIXED is close enough to STATEMENT to be OK.  If you read between the lines, temporary tables logged in statement form for both MIXED and STATEMENT, so switching between those is not a problem even with open temporary tables.

I think the safest course would be:
 on all of addons1..6
  SLAVE STOP;
  SET GLOBAL binlog_format = 'MIXED';
    (no change on addons1)
 on addons7
  SET GLOBAL binlog_format = 'MIXED';
 on addons1
  SLAVE START;
  SHOW SLAVE STATUS \G
 on addons2
  SLAVE START;
  SHOW SLAVE STATUS \G
 on addons3..6
  SLAVE START;
  SHOW SLAVE STATUS \G

----

As for the innodb_flush_method stuff -- I think that's a puppet bug.  It looks like, on a server's first startup, the puppet manifests enable the write cache, but configure MySQL without O_DIRECT.  I fixed that up in r48458.  Unfortunately, that leaves us with a five systems that aren't fully utilizing their hardware until the daemons are restarted.

But hey, that's what this bug is here to learn, right?

----

I'll wait for the restart of tp-bugs01-slave01 when it becomes a slave again.  If there's a bug for that, let's put a reminder to restart in there.

----

As for ft_min_word_len, I think I'll just ignore that permanently in both the crontask and the nagios check for now, and open a bug to remove the ignore in a few months, when hopefully most of these systems have restarted.

----

So here's the list:

Restart for O_DIRECT:
 dev1.db.scl3.mozilla.com (MASTER)
 dev2.db.scl3.mozilla.com
 addons1.stage.db.phx1.mozilla.com (MASTER)
 b1-db1.db.scl3.mozilla.com (MASTER/MASTER)
 engagement2.db.phx1.mozilla.com
 bedrock1.db.scl3.mozilla.com (MASTER)

Binlog update:
 addons{2..7).db.phx1.mozilla.com

RPM futzing:
 b2-db1.db.scl3.mozilla.com

Now, to find a time when sheeri's near a keyboard in case things go south..
I forgot in the above:

Restart for new basedir:
  node36.seamicro.phx1.mozilla.com


Many of the (RUN) above resulted in other servers being out of whack:

Variable                  /etc/my.cnf bedrock2.db.scl3.mozilla.com
========================= =========== ============================
expire_logs_days          0           10
wait_timeout              820         600
Variable                  /etc/my.cnf getpersonas1.db.scl3.mozilla.com
========================= =========== ================================
max_connections           1500        1200
Variable                  /etc/my.cnf db1.stage.bugs.scl3.mozilla.com
========================= =========== ===============================
innodb_flush_log_at_tr... 2           0
Variable                  /etc/my.cnf mysql1.stage.metrics.scl3.mozilla.com
========================= =========== =====================================
wait_timeout              14400       14401
Variable                  /etc/my.cnf mysql1.metrics.scl3.mozilla.com
========================= =========== =====================================
wait_timeout              14400       14401

..all resolved in favor of /etc/my.cnf.
dev cluster is restarted

b1-db1 ne devtools1 is restarted (with a puppet fix to generate the proper node id)
  (node 1 was the only active node when I found this cluster, so I left it that way)
  (this had some duplicate IDs, too - bug 796936)

engagement2 is restarted

addons1.stage is restarted

That leaves:

Restart for O_DIRECT:
 bedrock1.db.scl3.mozilla.com (MASTER)

Binlog update:
 addons{2..7).db.phx1.mozilla.com

RPM futzing:
 b2-db1.db.scl3.mozilla.com

Restart for new basedir:
  node36.seamicro.phx1.mozilla.com
b1-db1 restart's duplicate id's caused mayhem - bug 796991.  Le sigh.
bedrock1 restarted - it seems like nothing's using that cluster, as it has no databases.

b2-db1 is permanently horked.  It will just need to be re-built.  It doesn't match b2-db2 either..

node36 is part of the 'dtpt', which is engagement.  db-engagement-dev-rw has no failover set up, so I'm not keen to restart that master.
node36's *configuration* is wrong (in puppet).  The host itself is fine.  I fixed that, and it's good to go.  So just the binlog update to do here, then we can turn this into a nagios alert.
Can you define "permanently horked"? the database on b2-db1 seems to be working, right? What exactly is broken?
I can't install percona-toolkit due to dependency conflicts.  I force-installed some things, but the only perl-DBD-MySQL available is 3.007, which doesn't work with percona-toolkit anyway.  So it will fail this check (since pt-config-diff isn't installed) until we fix or upgrade it.
Ah! OK.
FWIW 3.007 works with pt-table-checksum no problem, on RHEL5 on tp-bouncer01-master01.
done comment 10 on addons1-7.db.phx
Thanks to sheeri's work on addons, and https://mana.mozilla.org/wiki/display/SYSADMIN/MySQL+5.0+to+5.1+Upgrade, all of the crontasks are working now, too, and diffs are mostly resolved.  

Outstanding questions are

1 config difference
Variable                  /etc/my.cnf plugins2.db.phx1.mozilla.com
========================= =========== ============================
query_cache_size          524288000   1048576000

1 config difference
Variable                  /etc/my.cnf tp-bugs01-slave01.phx.mozilla.com
========================= =========== =================================
innodb_buffer_pool_size   23622320128 19327352832

which should be pretty easy to solve (I'm just not sure which is right in each case).  I'll start working on a nagios check for this.
So, for plugins2, see https://bugzilla.mozilla.org/show_bug.cgi?id=798756#c36 - we turned on the query cache to try to see if caching would work. I made it larger when I saw low memory prunes.

And Qcache_lowmem_prunes is still increasing, so I doubled the size again today - it's now 2000M. This is actually OK to "alert" and make a bug about tuning, so please leave it as is, and let's make a bug to watch it and tune it.

As for the innodb_buffer_pool_size on tp-bugs01-slave01, the discrepancy is that it *was* a slave, and the slaves are set to 18G of buffer pool, but we switched it to be the master, which has 22G in /etc/my.cnf - but we never restarted it. This is, I think, legacy - I think at one point the master had 36G RAM and the slaves had 24G, but now they all have 36G.

So the answer is we should change all the tp-bugs01 slaves to be 22G, and tp-bugs01-slave01 will be a problem until we upgrade to MySQL 5.5 on it, which is scheduled for this quarter. So again, this is a legitimate "concern".
OK, cool.  I filed bug 803113 and bug 803114 to get those hosts to where they *should* be, and in the interim changed puppet to represent where they *are*, with a comment pointing to the relevant bug.

Nagios check, here we come.
Nagios check is in place.  This is only for mysql2-enabled servers, though.

> mysql1.metrics.scl3.mozilla.com:MySQL Config Diffs is CRITICAL: CRITICAL: 3 vars differ: innodb_flush_log_at_tr... innodb_flush_method sync_binlog (http://m.allizom.org/MySQL+Config+Diffs)
 -- acked for bug 803731
> tp-bugs01-slave01.phx.mozilla.com:MySQL Config Diffs is CRITICAL: CRITICAL: 1 vars differ: innodb_buffer_pool_size (http://m.allizom.org/MySQL+Config+Diffs)
 - dt'd for bug 803114
> b2-db2.db.scl3.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> datazilla1.db.scl3.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> datazilla2.db.scl3.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> dp-geodns01.phx.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> geodns1.vips.scl3.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> support1.db.phx1.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> support2.db.phx1.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> support3.db.phx1.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> support4.db.phx1.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> tp-a01-master01.phx.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> tp-a01-slave01.phx.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> tp-a01-slave02.phx.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> tp-b02-master01.phx.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> tp-c01-master01.phx.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
> tp-c01-slave01.phx.mozilla.com:MySQL Config Diffs is CRITICAL: NRPE: Command check_mysql_config_diffs not defined (http://m.allizom.org/MySQL+Config+Diffs)
 -- not using mysql2, so no check
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.