1131063 - Multiple newrelic agents not reporting

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

https://rpm.newrelic.com/accounts/677903/servers?showHidden=true

treeherder-rabbitmq1.stage.private.scl3.mozilla.com
treeherder-processor1.stage.private.scl3.mozilla.com
treeherder-processor2.stage.private.scl3.mozilla.com
treeherder-processor3.stage.private.scl3.mozilla.com
treeherder-etl1.stage.private.scl3.mozilla.com
treeherder-etl2.stage.private.scl3.mozilla.com

No data reporting for this server since Jan 29, 2015 3:03 PM.

Comment 1

[Kendall Libby [:fubar] (he/him)]

It's the datacenter proxy again; will go re-open that bug once I find it.

Comment 2

[Ed Morley [:emorley]]

Ah thank you :-)

Comment 3

[Ed Morley [:emorley]]

Comment 0 was about the nrsysmond agent not reporting for stage. 

In addition we're now seeing the python agents on multiple nodes on stage/prod stop reporting too. ie the lists of nodes are shrinking here:
https://rpm.newrelic.com/accounts/677903/applications/4180461/environment
https://rpm.newrelic.com/accounts/677903/applications/5585473/environment

Comment 4

[Ed Morley [:emorley]]

Attachment: run_celery_worker_buildapi-supervisor.log

From treeherder-etl1.private.scl3.

The "UserWarning: Cannot load extension u'celerymon.bin.celerymon:MonitorDelegate'" appears earlier in the log, so seems unrelated.

However what is noticeably absent from the log after the last process restart is _any_ newrelic agent output at all. It's like we're not even running it.

Comment 5

[Ed Morley [:emorley]]

Yeah if you look at the exception tracebacks (since they reveal the python version), the newrelic agent output seems to stop immediately after the process restart where python 2.7 was deployed.

Comment 7

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #5)
> Yeah if you look at the exception tracebacks (since they reveal the python
> version), the newrelic agent output seems to stop immediately after the
> process restart where python 2.7 was deployed.

We're getting no reporting at all now on prod (https://rpm.newrelic.com/accounts/677903/applications/4180461) could you take a look? :-)

Comment 8

[Kendall Libby [:fubar] (he/him)]

It's on infra/opsec to fix, in 1114611.

Comment 9

[Kendall Libby [:fubar] (he/him)]

newrelic agents added to python27 and jobs restarted. looks like things are showing up in new relic again. even made the warning banner about mismatched versions go away.

Comment 10

[Mauro Doglio [:mdoglio]]

Thanks :fubar!