Closed Bug 930585 Opened 11 years ago Closed 11 years ago

remove statsd from socorro dev, stage, prod config

Categories

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Socorro
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lonnen, Assigned: bburton)

Details

Attachments

(1 file)

Recent socorro metrics
21.30 KB, text/plain
Details 
Stop sending any metrics to statsd/graphite for a while.
Assignee: server-ops-webops → bburton
Commented out locally for dev

[root@socorro1.dev.dmz.phx1 socorro]# grep -R statsd * | grep graphite
common.conf:#export statsdHost=graphite1.private.phx1.mozilla.com
common_statsd.ini:#statsd_host='graphite1.private.phx1.mozilla.com'
Committed commented out configs for stage

-> % svn diff
Index: common.conf
===================================================================
--- common.conf	(revision 76983)
+++ common.conf	(working copy)
@@ -55,7 +55,7 @@

 export product='Firefox,Fennec,FennecAndroid'
 # bug 790097
-export statsdHost=graphite1.dmz.phx1.mozilla.com
+#export statsdHost=graphite1.dmz.phx1.mozilla.com
 export statsdPrefix=socorro-stage
 # bug 822828
 export brokenBoot2GeckoFixer=/data/bin/minidump_hack-b2g
bburton@althalus [09:28:56] [~/code/mozilla/sysadmins/puppet/trunk/modules/socorro/files/stage/etc-socorro]
-> % svn ci -m "commenting out statsd config for socorro stage, bug 930585"
export database_username='breakpad_rw'
Sending        etc-socorro/common.conf
Transmitting file data .
Committed revision 76985.
Committed commented out configs for prod

-> % svn diff
Index: admin_common.conf
===================================================================
--- admin_common.conf	(revision 76983)
+++ admin_common.conf	(working copy)
@@ -48,7 +48,7 @@

 export product='Firefox,Fennec,FennecAndroid'

-export statsdHost=graphite1.dmz.phx1.mozilla.com
+#export statsdHost=graphite1.dmz.phx1.mozilla.com
 export statsdPrefix=socorro-prod

 # bug 822828
Index: common_statsd.ini
===================================================================
--- common_statsd.ini	(revision 76983)
+++ common_statsd.ini	(working copy)
@@ -12,7 +12,7 @@
 # name: statsd_host
 # doc: the hostname of statsd
 # converter: str
-statsd_host='graphite1.private.phx1.mozilla.com'
+#statsd_host='graphite1.private.phx1.mozilla.com'

 # name: statsd_port
 # doc: the port number for statsd
Index: common.conf
===================================================================
--- common.conf	(revision 76983)
+++ common.conf	(working copy)
@@ -49,7 +49,7 @@

 export product='Firefox,Fennec,FennecAndroid'

-export statsdHost=graphite1.dmz.phx1.mozilla.com
+#export statsdHost=graphite1.dmz.phx1.mozilla.com
 export statsdPrefix=socorro-prod

 # bug 822828
bburton@althalus [09:30:36] [~/code/mozilla/sysadmins/puppet/trunk/modules/socorro/files/prod/etc-socorro]
-> % svn ci -m "commenting out statsd config for socorro prod, bug 930585"
Sending        etc-socorro/admin_common.conf
Sending        etc-socorro/common.conf
Sending        etc-socorro/common_statsd.ini
Transmitting file data ...
Committed revision 76986.
bburton@althalus [09:49:24] [~/code/mozilla/sysadmins/puppet/trunk/modules/socorro]
-> % svn diff
Index: templates/stage/etc-socorro/local.py.erb
===================================================================
--- templates/stage/etc-socorro/local.py.erb	(revision 76917)
+++ templates/stage/etc-socorro/local.py.erb	(working copy)
@@ -21,8 +21,8 @@
 CORRELATION_SERVER = 'https://crash-stats.allizom.org'

 # for local development these don't matter
-STATSD_HOST = 'graphite1.private.phx1.mozilla.com'
-STATSD_PREFIX = 'socorro-stage.webapp'
+#STATSD_HOST = 'graphite1.private.phx1.mozilla.com'
+#STATSD_PREFIX = 'socorro-stage.webapp'

 CACHES = {
     'default': {
bburton@althalus [09:49:26] [~/code/mozilla/sysadmins/puppet/trunk/modules/socorro]
-> % svn ci -m "commenting out statsd config for socorro stage, bug 930585"
Sending        socorro/templates/stage/etc-socorro/local.py.erb
Transmitting file data .
Committed revision 76988.
Pushed to prod django and apache restarted

[root@socorroadm.private.phx1 ~]# cp /data/crashstats/src/crash-stats.mozilla.org/local.py /data/crashstats/src/crash-stats.mozilla.org/socorro/webapp-django/crashstats/settings/local.py
cp: overwrite `/data/crashstats/src/crash-stats.mozilla.org/socorro/webapp-django/crashstats/settings/local.py'? y
[root@socorroadm.private.phx1 ~]# /data/crashstats/deploy crash-stats.mozilla.org[2013-10-24 17:00:07] Running rsync_project
[2013-10-24 17:00:07] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/
[2013-10-24 17:00:08] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/ (0.554s)
[2013-10-24 17:00:08] Finished rsync_project (0.554s)
[2013-10-24 17:00:08] Running commit_www
[2013-10-24 17:00:08] [localhost] running: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']'
[2013-10-24 17:00:08] [localhost] finished: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']' (0.394s)
[localhost] out: [master f64b13b] deploy [crash-stats.mozilla.org]
[localhost] out: 1 file changed, 2 insertions(+), 2 deletions(-)
[2013-10-24 17:00:08] Finished commit_www (0.395s)
[2013-10-24 17:00:08] Running push_www
[2013-10-24 17:00:08] [socorro1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-10-24 17:00:08] [socorro2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-10-24 17:00:08] [socorro3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-10-24 17:00:08] [socorro4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-10-24 17:00:09] [socorro1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (0.840s)
[socorro1.webapp.phx1.mozilla.com] out: Not removing crash-stats.mozilla.org/app/
[2013-10-24 17:00:09] [socorro2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (0.946s)
[2013-10-24 17:00:09] [socorro3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (0.975s)
[2013-10-24 17:00:09] [socorro4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (0.972s)
[2013-10-24 17:00:09] Finished push_www (0.982s)
[root@socorroadm.private.phx1 ~]# issue-multi-command crashstats service httpd graceful
[2013-10-24 17:02:13] [socorro1.webapp.phx1.mozilla.com] running: service httpd graceful
[2013-10-24 17:02:13] [socorro2.webapp.phx1.mozilla.com] running: service httpd graceful
[2013-10-24 17:02:13] [socorro3.webapp.phx1.mozilla.com] running: service httpd graceful
[2013-10-24 17:02:13] [socorro4.webapp.phx1.mozilla.com] running: service httpd graceful
[2013-10-24 17:02:13] [socorro1.webapp.phx1.mozilla.com] finished: service httpd graceful (0.389s)
[2013-10-24 17:02:13] [socorro3.webapp.phx1.mozilla.com] finished: service httpd graceful (0.411s)
[2013-10-24 17:02:13] [socorro2.webapp.phx1.mozilla.com] finished: service httpd graceful (0.458s)
[2013-10-24 17:02:13] [socorro4.webapp.phx1.mozilla.com] finished: service httpd graceful (0.470s)
Stage Django finally pushed and apache restarted

[root@socorroadm.private.phx1 ~]# issue-multi-command crashstats-stage service httpd restart
[2013-10-24 17:30:43] [socorro1.stage.webapp.phx1.mozilla.com] running: service httpd restart
[2013-10-24 17:30:43] [socorro2.stage.webapp.phx1.mozilla.com] running: service httpd restart
[2013-10-24 17:30:43] [socorro2.stage.webapp.phx1.mozilla.com] finished: service httpd restart (0.496s)
[socorro2.stage.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro2.stage.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-10-24 17:30:43] [socorro1.stage.webapp.phx1.mozilla.com] finished: service httpd restart (0.562s)
[socorro1.stage.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro1.stage.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Attached file Recent socorro metrics — 
These are the only socorro-related metrics I seem to be getting now, 218 of them.
(In reply to Eric Ziegenhorn :ericz from comment #7)
> Created attachment 822023 [details]
> Recent socorro metrics
> 
> These are the only socorro-related metrics I seem to be getting now, 218 of
> them.

I'm running a few tcpdumps like 'tcpdump host 10.8.75.93'

I did see statsd stuff going out from processor[1-2].stage and disabled the config there, and I no longer see it, I'm not seeing anything go out from the other hosts, does statsd need a kick?
Ah, that could be it.  It seemed to be persistently sending old metrics once before.  I'll dig into that, thanks.
I left all those tcpdumps running overnight and didn't see any packets sent
I believe it likely was statsd.  I found that there is an option in newish versions to stop sending metrics when it stops receiving them and will look into that.
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: