783078 - graphite support for releng

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

web2.releng.webapp.scl3 and relengwebadm.private.scl3 do not appear to be in ganglia.  Please add them so we'll have dataz when we need them.  Thanks!

Comment 1

[Ashish Vijayaram [:ashish]]

Looks like this needs flows from each of the servers to ganglia1.dmz.scl3.mozilla.com:8649. Presently only relengweb1 is in the cluster but hte configs are already added for to web1, web2, adm - https://ganglia-scl3.mozilla.org/ganglia/?c=RelEngWeb

Comment 2

[Rick Bryce [:rbryce]]

Moving these nodes to ganglia1.private.scl3.m.c (bug 775517).  Then I will I get the flows hammered out.

Comment 3

[Shyam Mani [:fox2mike]]

Can this to go graphite now?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Sure.  It's still stalled on releng, so it's getting 0 hits and its preformance is great!!

Comment 5

[Rick Bryce [:rbryce]]

/cc :arr

I know you are still putting together the people from releng for getting collectd for releng systems.  In the mean time, please list what operating systems you know collectd will need to run on. I can get started finding/building collectd for systems we may not have.

Comment 6

[Amy Rich [:arr] [:arich]]

rbryce: I morphed the bug subject to be general support for graphite for releng since you asked for that in comment 5.  In addition to the various Redhat versions that infra is supporting for releng infrastructure, we would also like to add the following if we can find appropriate clients:

OS X 10.6
OS X 10.7
OS X 10.8
Windows XP
Windows 7
Windows 8
Windows 2008R2
CentOS 5
CentOS 6

As Mozilla adds new OSes to its Firefox test platform, we'll need to add those OSes, too (so probably OS X 10.9 when Apple releases it).

The most important OSes to start with (that we will try to roll out first) will be CentOS 6, OS X 10.7, and Windows 2008R2.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Rick, these will need to be installed via PuppetAgain
  https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain

I can help you with the setup for that.  Building packages is definitely the first step.  Then we'll need to get them installed on the puppetmasters, write up a patch to install and configure them, get that patch reviewed, and land it.  I assume we can do this in parallel with the existing Ganglia configuration, so the two can overlap for a while?

Comment 8

[Rick Bryce [:rbryce]]

(In reply to Dustin J. Mitchell [:dustin] from comment #7)
> Rick, these will need to be installed via PuppetAgain
>   https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain
> 
> I can help you with the setup for that.  Building packages is definitely the
> first step.  Then we'll need to get them installed on the puppetmasters,
> write up a patch to install and configure them, get that patch reviewed, and
> land it.  I assume we can do this in parallel with the existing Ganglia
> configuration, so the two can overlap for a while?

Yes, ganglia and collectd can run together with no issues.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

This bug was morphed, which may be confusing.  At this point, then plan is to run collectd on just about all releng systems, not just the releng cluster as originally described.

Comment 10

[Rick Bryce [:rbryce]]

We have provided the Graphite infrastructure. Relops has been doing the collectd plumbing over in Bug 870853.