971799 - Log # network bytes transmitted/received to Graphite

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John Hopkins (:jhopkins)]

We currently only log packets, not actual bytes transmitted/received.

Comment 1

[John Hopkins (:jhopkins)]

Attachment: [puppet] configure collectd::plugins::ethstat

Comment 2

[Jake Watkins [:dividehex]]

Comment on attachment 8374927 [details] [diff] [review]
[puppet] configure collectd::plugins::ethstat

This looks fine.  My only concern is it will add about 50 more metrics per node.  I would like to determine how many total new metrics this will add up to and double check with :ericz on how it might impact the graphite cluster.  When new metrics are added, carbon does rate limit the whisper db file creation process so it might also take a day or two for these new metrics to appear.

I don't exactly remember the total count of releng centos and ubuntu instances.

:jhopkins, would you know?

r+ if it won't bring down graphite :-)

Comment 3

[John Hopkins (:jhopkins)]

What I could do instead is "Map" the metrics I am interested in and limit reporting to just those metrics.  For example:

  Map "rx_bytes" "if_rx_bytes"
  Map "tx_bytes" "if_tx_bytes"
  Map "rx_errors" "if_rx_errors"
  Map "tx_errors" "if_tx_errors"
  MappedOnly true

Still would be good to know what impact logging all the metrics has.

The number of machines checking in varies.  There's ~1,050 AWS instances running at the moment, plus what physical Centos and Ubuntu machines we have in Inventory.

Comment 4

[Eric Ziegenhorn :ericz]

We certainly don't have room for 50 more metrics per host.  :jhopkins are you saying you could add just 4 more?  4 would work but be tight in SCL3 (relevant bug is 971389 to free up more space).  PHX1 is already overloaded so I'm not sure it really would matter and that's being worked on in bug 919038.

Comment 5

[John Hopkins (:jhopkins)]

I could use just two more metrics (rx_bytes, tx_bytes), even, since rx/tx errors are already covered by the Interface module.

Comment 6

[Eric Ziegenhorn :ericz]

Works for me.

Comment 7

[John Hopkins (:jhopkins)]

Attachment: [puppet] reduce list of metrics vs. original patch

Comment 8

[Jake Watkins [:dividehex]]

Comment on attachment 8375701 [details] [diff] [review]
[puppet] reduce list of metrics vs. original patch

Looks good.  Although I'm fairly certain ethstat isn't compatible with OSX.  Might want to take out the Darwin case so it fails if added to the darwin profile.

Comment 9

[John Hopkins (:jhopkins)]

Comment on attachment 8375701 [details] [diff] [review]
[puppet] reduce list of metrics vs. original patch

Removed 'Darwin' case and landed as 
https://hg.mozilla.org/build/puppet/rev/34372e5c9d6d

Comment 10

[John Hopkins (:jhopkins)]

Merged to production.  New ethstat/* metrics should show up in the next day or two (mentioned in comment 2) for Linux hosts.

Comment 11

[John Hopkins (:jhopkins)]

Unfortunately, this is only working for physical hosts.  EC2 hosts report no stats, and I've reproduced same from the console "ethtool -S eth0".

Comment 12

[John Hopkins (:jhopkins)]

Currently working on win64 related work which is a higher priority (per taras).  Unassigning so I don't block someone else working on this bug.