1157337 - Upgrade collectd to latest (5.5.0)

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Jake Watkins [:dividehex]]

Test and deploy the latest version of collectd.

Comment 1

[Jake Watkins [:dividehex]]

Build config output for 5.4.2 on yosemite:

Configuration:
  Libraries:
    intel mic . . . . . . no (MicAccessApi not found)
    libaquaero5 . . . . . no (libaquaero5.h not found)
    libcurl . . . . . . . yes
    libdbi  . . . . . . . no (dbi/dbi.h not found)
    libcredis . . . . . . no (credis.h not found)
    libesmtp  . . . . . . no (libesmtp not found)
    libganglia  . . . . . no (gm_protocol.h not found)
    libgcrypt . . . . . . no (gcrypt.h not found)
    libiokit  . . . . . . yes
    libiptc . . . . . . . no (Linux only)
    libjvm  . . . . . . . no (jar not found)
    libkstat  . . . . . . no (Solaris only)
    libkvm  . . . . . . . no
    libmemcached  . . . . no (libmemcached/memcached.h not found)
    libmnl  . . . . . . . no (Linux only library)
    libmodbus . . . . . . no (pkg-config doesn't know libmodbus)
    libmysql  . . . . . . yes
    libnetapp . . . . . . no (netapp_api.h not found)
    libnetsnmp  . . . . . yes
    libnotify . . . . . . no (No package 'libnotify' found)
    liboconfig  . . . . . yes (shipped version)
    libopenipmi . . . . . no (pkg-config doesn't know OpenIPMIpthread)
    liboping  . . . . . . no (oping.h not found)
    libpcap . . . . . . . yes
    libperfstat . . . . . no (AIX only)
    libperl . . . . . . . yes (version 5.18.2)
    libpq . . . . . . . . no (libpq-fe.h not found)
    libpthread  . . . . . yes
    librabbitmq . . . . . no (amqp.h not found)
    librouteros . . . . . no (routeros_api.h not found)
    librrd  . . . . . . . no (rrd.h not found)
    libsensors  . . . . . no (Linux only library)
    libsigrok   . . . . . no (libsigrok/libsigrok.h not found)
    libstatgrab . . . . . no (pkg-config doesn't know libstatgrab)
    libtokyotyrant  . . . no (tcrdb.h not found)
    libupsclient  . . . . no (pkg-config doesn't know libupsclient)
    libvarnish  . . . . . no (pkg-config doesn't know varnishapi)
    libvirt . . . . . . . no (pkg-config doesn't know libvirt)
    libxml2 . . . . . . . yes
    libxmms . . . . . . . no
    libyajl . . . . . . . no (yajl/yajl_parse.h not found)
    libevent  . . . . . .
    protobuf-c  . . . . . no
    oracle  . . . . . . . no (ORACLE_HOME is not set)
    python  . . . . . . . no

  Features:
    daemon mode . . . . . yes
    debug . . . . . . . . no

  Bindings:
    perl  . . . . . . . . yes (PREFIX=/usr/local)

  Modules:
    aggregation . . . . . yes
    amqp    . . . . . . . no
    apache  . . . . . . . yes
    apcups  . . . . . . . yes
    aquaero . . . . . . . no
    apple_sensors . . . . yes
    ascent  . . . . . . . yes
    battery . . . . . . . yes
    bind  . . . . . . . . yes
    conntrack . . . . . . no
    contextswitch . . . . yes
    cgroups . . . . . . . no
    cpu . . . . . . . . . yes
    cpufreq . . . . . . . no
    csv . . . . . . . . . yes
    curl  . . . . . . . . yes
    curl_json . . . . . . no
    curl_xml  . . . . . . yes
    dbi . . . . . . . . . no
    df  . . . . . . . . . yes
    disk  . . . . . . . . yes
    dns . . . . . . . . . yes
    email . . . . . . . . yes
    entropy . . . . . . . no
    ethstat . . . . . . . no
    exec  . . . . . . . . yes
    filecount . . . . . . yes
    fscache . . . . . . . no
    gmond . . . . . . . . no
    hddtemp . . . . . . . yes
    interface . . . . . . yes
    ipmi  . . . . . . . . no
    iptables  . . . . . . no
    ipvs  . . . . . . . . no
    irq . . . . . . . . . no
    java  . . . . . . . . no
    libvirt . . . . . . . no
    load  . . . . . . . . yes
    logfile . . . . . . . yes
    lpar  . . . . . . . . no
    lvm . . . . . . . . . no
    madwifi . . . . . . . no
    match_empty_counter . yes
    match_hashed  . . . . yes
    match_regex . . . . . yes
    match_timediff  . . . yes
    match_value . . . . . yes
    mbmon . . . . . . . . yes
    md  . . . . . . . . . no
    memcachec . . . . . . no
    memcached . . . . . . yes
    memory  . . . . . . . yes
    mic . . . . . . . . . no
    modbus  . . . . . . . no
    multimeter  . . . . . yes
    mysql . . . . . . . . yes
    netapp  . . . . . . . no
    netlink . . . . . . . no
    network . . . . . . . yes
    nfs . . . . . . . . . no
    nginx . . . . . . . . yes
    notify_desktop  . . . no
    notify_email  . . . . no
    ntpd  . . . . . . . . yes
    numa  . . . . . . . . no
    nut . . . . . . . . . no
    olsrd . . . . . . . . yes
    onewire . . . . . . . no
    openvpn . . . . . . . yes
    oracle  . . . . . . . no
    perl  . . . . . . . . yes
    pf  . . . . . . . . . no
    pinba . . . . . . . . no
    ping  . . . . . . . . no
    postgresql  . . . . . no
    powerdns  . . . . . . yes
    processes . . . . . . yes
    protocols . . . . . . no
    python  . . . . . . . no
    redis . . . . . . . . no
    routeros  . . . . . . no
    rrdcached . . . . . . no
    rrdtool . . . . . . . no
    sensors . . . . . . . no
    serial  . . . . . . . no
    sigrok  . . . . . . . no
    snmp  . . . . . . . . yes
    statsd  . . . . . . . yes
    swap  . . . . . . . . yes
    syslog  . . . . . . . yes
    table . . . . . . . . yes
    tail  . . . . . . . . yes
    tail_csv  . . . . . . yes
    tape  . . . . . . . . no
    target_notification . yes
    target_replace  . . . yes
    target_scale  . . . . yes
    target_set  . . . . . yes
    target_v5upgrade  . . yes
    tcpconns  . . . . . . yes
    teamspeak2  . . . . . yes
    ted . . . . . . . . . yes
    thermal . . . . . . . no
    threshold . . . . . . yes
    tokyotyrant . . . . . no
    unixsock  . . . . . . yes
    uptime  . . . . . . . yes
    users . . . . . . . . yes
    uuid  . . . . . . . . yes
    varnish . . . . . . . no
    vmem  . . . . . . . . no
    vserver . . . . . . . no
    wireless  . . . . . . no
    write_graphite  . . . yes
    write_http  . . . . . yes
    write_mongodb . . . . no
    write_redis . . . . . no
    write_riemann . . . . no
    xmms  . . . . . . . . no
    zfs_arc . . . . . . . no

Comment 2

[Jake Watkins [:dividehex]]

Versions for 10.7, 10.8 and 10.10 have been built and uploaded to releng-puppet2.srv.releng.  I'll need to find a 10.6.8 host with xcode installed or I need to get an install disk and install a vm in Fusion to build it.

As for 10.9, I compiled a build but my VM seems to be missing libiokit which is causing it to skip building the disk module.  I'll need to fix that since the disk module is a must have.

Comment 3

[Jake Watkins [:dividehex]]

10.6 version built and uploaded

Comment 4

[Jake Watkins [:dividehex]]

For 10.9, I reinstalled xcode and rebuilding it found libiokit.

Comment 5

[Jake Watkins [:dividehex]]

Looks like 5.5.0 was release last week which includes the ability for the cpu module to report percentage instead of jiffies.  Since I was half way through backporting 5.4 to precise, I've switched gears to 5.5.0.

It should be trivial to fire off the builds for OSX and centos.

Comment 6

[Jake Watkins [:dividehex]]

Attachment: bug1157337-1.patch

Adds collectd custom repos but does not bump installed version yet

Comment 7

[Jake Watkins [:dividehex]]

Comment on attachment 8620502 [details] [diff] [review]
bug1157337-1.patch

remote:   https://hg.mozilla.org/build/puppet/rev/4072c3b67541
remote:   https://hg.mozilla.org/build/puppet/rev/675660d42cf1

Comment 8

[Jake Watkins [:dividehex]]

Attachment: bug1157337-2.patch

* Removes unused centos collectd package modules (cleanup)
* Ensure absent -dev and -dbg ubuntu collectd package modules
* Bumps collectd to version 5.5.0 across all posix platforms

Tested on Centos 6.5, Precise, Trusty and OSX 10.6

Comment 9

[Jake Watkins [:dividehex]]

Comment on attachment 8620742 [details] [diff] [review]
bug1157337-2.patch

Ignore the node change at slipped in

Comment 10

[Amy Rich [:arr] [:arich]]

It looks like the addition to the ubuntu repo caused some issues with the AWS puppet runs for golden images last night. Should probably give you a chance to check into that before I do the next review.

Comment 11

[Jake Watkins [:dividehex]]

Comment on attachment 8620742 [details] [diff] [review]
bug1157337-2.patch

Checked in less the node def

remote:   https://hg.mozilla.org/build/puppet/rev/7407874b0a9c
remote:   https://hg.mozilla.org/build/puppet/rev/e8b6fadd9334

Comment 12

[Jake Watkins [:dividehex]]

Attachment: bug1157337-backout.patch

Looks like cpu module for OSX is broken in 5.5.0 :-/

http://mailman.verplant.org/pipermail/collectd/2015-May/006547.html
https://github.com/collectd/collectd/commit/e0683047a42e217c352c2419532b8e029f9f3f0a

This patch downgrades collectd back to 5.3.0 on Darwin only

Jun 11 03:08:00 r4-mini-001 collectd[46412]: cpu plugin: processor_info returned only 4 elements..
Jun 11 03:13:00 r4-mini-001 collectd[46412]: cpu plugin: processor_info returned only 4 elements..
Jun 11 03:13:00 r4-mini-001 collectd[46412]: cpu plugin: processor_info returned only 4 elements..

Comment 13

[Jake Watkins [:dividehex]]

Comment on attachment 8621180 [details] [diff] [review]
bug1157337-backout.patch

Once again, ignore the node def

Comment 14

[Jake Watkins [:dividehex]]

Comment on attachment 8621180 [details] [diff] [review]
bug1157337-backout.patch

Less the node def that I can never seem to remember to exclude from my patches
remote:   https://hg.mozilla.org/build/puppet/rev/a3cccbd455d1
remote:   https://hg.mozilla.org/build/puppet/rev/fd6d25a8973e

Comment 15

[Jake Watkins [:dividehex]]

Attachment: bug1157337-4.patch

Build script and cpu states patch to build 5.5.0 for OSX

Comment 16

[Amy Rich [:arr] [:arich]]

Comment on attachment 8621815 [details] [diff] [review]
bug1157337-4.patch

Reading an hg diff of a git diff temporarily broke my brain. :}

My only feedback is that I notice that the only other patch file we store in that directory is just the bare patch, not a mail message containing the patch. I don't know if we want that to remain consistent one way or the other.

Comment 17

[Jake Watkins [:dividehex]]

(In reply to Amy Rich [:arr] [:arich] from comment #16)
> Comment on attachment 8621815 [details] [diff] [review]
> bug1157337-4.patch
> 
> Reading an hg diff of a git diff temporarily broke my brain. :}
> 
> My only feedback is that I notice that the only other patch file we store in
> that directory is just the bare patch, not a mail message containing the
> patch. I don't know if we want that to remain consistent one way or the
> other.

I don't think it really matters.  I prefer to keep the full patch with comments for context.  This entire commit is simple for anyone in the future who is looking to build collectd.  The more context the better imho.

Comment 18

[Jake Watkins [:dividehex]]

Comment on attachment 8621815 [details] [diff] [review]
bug1157337-4.patch

remote:   https://hg.mozilla.org/build/puppet/rev/1b51cb194fb2
remote:   https://hg.mozilla.org/build/puppet/rev/0b995ab1e350

Comment 19

[Jake Watkins [:dividehex]]

Attachment: bug1157337-5.patch

I've rebuilt the osx dmgs with the cpu states patch.  This bumps the version to 5.5.0-1

Comment 20

[Amy Rich [:arr] [:arich]]

(In reply to Jake Watkins [:dividehex] from comment #17)
I agree that having more context is better. I was wondering if we had that for the other patch.

Comment 21

[Jake Watkins [:dividehex]]

Comment on attachment 8621865 [details] [diff] [review]
bug1157337-5.patch

remote:   https://hg.mozilla.org/build/puppet/rev/ea1805cfd117
remote:   https://hg.mozilla.org/build/puppet/rev/202efb9c5de3