778831 - Setup tcpdump on socorro staging processors, hbase, and postgres

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Brandon Burton [:solarce]]

Tracking bug for Setup tcpdump on socorro staging processors, hbase, and postgres with a rolling window to get some packet captures

Comment 1

[Brandon Burton [:solarce]]

Postgres has been stable since we tweaked the connection time out setting.

Working on tcpdumps for processors, monitor, and hbase nodes.

Comment 2

[Brandon Burton [:solarce]]

tcpdump -Z root -i any -s0 -C 5000000 -W 1000 -w socorro.778831 is now running on supportadm.stage, processor1-2, and hp-node62-69

Comment 3

[Ashish Vijayaram [:ashish]]

(In reply to Brandon Burton [:solarce] from comment #2)
> tcpdump -Z root -i any -s0 -C 5000000 -W 1000 -w socorro.778831 is now
> running on supportadm.stage, processor1-2, and hp-node62-69
...and filling up disk by a significant extent on socorro-processor2.stage.metrics.phx1. I'm gonna gzip some older logs.

Comment 4

[Adrian J Fernandez [:Aj]]

gzip logs older than 3 days (128 files) on socorro-processor1.stage.metrics.phx1 as well as;
Before
177G    /root/778831/

After
114G    /root/778831/

Command ran;
cd /root/778831/
find . -atime +3 | xargs gzip

Comment 5

[Brandon Burton [:solarce]]

I have stopped all the tcpdumps that were running and deleted all the pcap files to clean things up.