885332 - Received Page when not oncall

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Peter ONeill [:zero]]

for some reason I received a page even though inventory says that jbraddock is on call. 

from 1(650)-946-8510

nagios1.private.scl3.mozilla.com:SN P3 is WARNING: WARNING ISSUES:(INC0011527)

Comment 1

[Ashish Vijayaram [:ashish]]

Per logs, oncall changed from zero to jbraddock at Wed Jun 19 10:10:01 PDT 2013.

Nagios sent alerts about INC0011527 at these times:

Thu Jun 20 05:41:08 PDT 2013
Thu Jun 20 05:51:08 PDT 2013
Thu Jun 20 06:01:08 PDT 2013

Looks consistent.

Comment 2

[Joel Braddock :jbraddock]

I have not received a text notifying me that I am now on call - I always receive this text upon change. I also have not received any texts reg. INC0011527 at any of the times listed in Comment 1. 

Can I get someone to check the setup once again?

Comment 3

[Ashish Vijayaram [:ashish]]

(In reply to Joel Braddock :jbraddock from comment #2)
> I have not received a text notifying me that I am now on call - I always
> receive this text upon change. I also have not received any texts reg.
> INC0011527 at any of the times listed in Comment 1. 
> 

I've fixed the oncall change notifications. Sorry about that. Re-reading Comment 1, I realise I hadn't had coffee.

> Can I get someone to check the setup once again?

Sure. I'll work with you on IRC. Please ping me when you have a few mins. Thanks!

Comment 4

[Ashish Vijayaram [:ashish]]

(In reply to Ashish Vijayaram [:ashish] from comment #3)
> > Can I get someone to check the setup once again?
> 
> Sure. I'll work with you on IRC. Please ping me when you have a few mins.
> Thanks!

I tracked down this bug to Bug 881306, where I had missed switching over the desktop oncall contact in Nagios. I can now correlate and confirm :zero received said pages. I've pushed out a fix now. Please reopen this bug or feel free to ping me in IRC if you still see issues.

Sorry for the trouble!

Comment 5

[Joel Braddock :jbraddock]

Testing w/ INC0011535 and I received pages.

Comment 6

[Joel Braddock :jbraddock]

Hey guys - I am receiving pings and I am not on call. hlangi should be receiving the pages. Can you please check the config again?

Comment 7

[Ashish Vijayaram [:ashish]]

(In reply to Joel Braddock :jbraddock from comment #6)
> Hey guys - I am receiving pings and I am not on call. hlangi should be
> receiving the pages. Can you please check the config again?

What was the page you received?

Comment 8

[Joel Braddock :jbraddock]

I received a text at 8:37PM PST on July 2nd that stated:

"PROBLEM (WARNING) SN PC on nagios1.private.scl3.mozilla.com: WARNING ISSUES:(INC0011584)"

Comment 9

[Ashish Vijayaram [:ashish]]

As per logs, oncall changed from pdang on July 3 08:25. Here are the last few oncall switches. Please clarify the times:

desktop oncall set to zero at Mon Jun 17 04:55:02 PDT 2013
desktop oncall changed to jbraddock at Wed Jun 19 10:10:01 PDT 2013
desktop oncall changed to hlangi at Wed Jun 26 10:40:01 PDT 2013
desktop oncall changed to jbraddock at Thu Jun 27 02:00:02 PDT 2013
desktop oncall changed to pdang at Wed Jul 3 08:25:02 PDT 2013

Comment 10

[Joel Braddock :jbraddock]

Interesting - is there any way to see who made this change?

"desktop oncall changed to jbraddock at Thu Jun 27 02:00:02 PDT 2013"

Comment 11

[Ashish Vijayaram [:ashish]]

(In reply to Joel Braddock :jbraddock from comment #10)
> Interesting - is there any way to see who made this change?
> 
> "desktop oncall changed to jbraddock at Thu Jun 27 02:00:02 PDT 2013"

Unfortunately no :(

Comment 12

[Joel Braddock :jbraddock]

OK - we will keep an eye on it the following weeks. Closing this out for now.