822415 - Change the monitor for Socorro replication to use hot_standby_delay instead of replicate_row

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Selena Deckelmann :selenamarie :selena]

We had a problem where the check returned a false positive.

:dumitru is working on changing this.

Comment 1

[Selena Deckelmann :selenamarie :selena]

There is a bug in the perl script that doesn't properly handle '--port2' and '--host2' parameters as you'd expect from the documentation.

If you specify this like so: 

--port=5432,5432
--host=host1,host2

That should fix the error that we saw on Tuesday.

Comment 2

[Dumitru Gherman [:dumitru]]

[[e-mail]1 mozilla]# ./check_postgres.pl --host=tp-socorro01-master01.phx1.mozilla.com,tp-socorro01-master02.phx1.mozilla.com --dbuser=nagiosdaemon --dbname=breakpad --action=hot_standby_delay  --dbuser2=nagiosdaemon --dbname2=breakpad  --warning=10 --critical=20
Password for user nagiosdaemon:
Password for user nagiosdaemon:
Password for user nagiosdaemon:
Password for user nagiosdaemon:
POSTGRES_HOT_STANDBY_DELAY OK: DB "breakpad" (host:tp-socorro01-master01.phx1.mozilla.com) -25328 | time=0.42s replay_delay=-25328;10;20  receive-delay=-25328;10;20
[[e-mail]1 mozilla]# ./check_postgres.pl -V
check_postgres.pl version 2.19.0


So, remember when I first tried the hot_standby_delay check with the older version of the script? It returned the same big negative values.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Yeah, it's because the standby receives WAL between the time the check runs on the master (first) and the check runs on the replica (second).

It's not beautiful, but it does accurately represent the state of the system. They're looking at changing the logic for the script to ask the replica for it's location first.

Comment 4

[Dumitru Gherman [:dumitru]]

I see.
Does this mean we can switch to this check? If so, we need to fine tune it to alert us when the deltas are too high.

Comment 5

[Selena Deckelmann :selenamarie :selena]

Let's try setting the delta to warn at 16777216 (that's in bytes, 16 MB). It should never get that high if things are working.

Comment 6

[Selena Deckelmann :selenamarie :selena]

ping!

Comment 7

[Dumitru Gherman [:dumitru]]

Completed:

[10:05] <nagios-phx1> | dumitru: tp-socorro01-master02.phx1.mozilla.com:PostgreSQL Hot Standby Delay is OK - POSTGRES_HOT_STANDBY_DELAY OK: (host:tp-socorro01-master01.phx1.mozilla.com =>
                      tp-socorro01-master02.phx1.mozilla.com) 0 Last Checked: 2013-01-04 10:02:41 PST


So I replaced the "replicate_row" with "hot_standby_delay".
Thresholds are: warning at 16777216 and critical at 33554432.