@wwwLocations = ("/mnt/data/stats/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp3.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/"); No logs for ftp2.dmz.scl3.mozilla.com and ftp5.dmz.scl3.mozilla.com since 5/14.
Eric, I re-built and re-puppetized these machines, can you look into what was missed here and add it into puppet?
It's not compressing the logs, so they never get shipped. The compression script appears to have been killed a year ago so I'm surprised we're just now running in to this. Looking into integrating the compression into rsync_file.sh.
I noticed it while making changes for the CDH upgrade. I assume there's no way to get the data for earlier days? Can we have some checks in place to ensure this doesn't happen again? -anurag
Yes, the data from earlier days is still there and is on metrics-logger1 now. We do have checks in place for this but this was lost in the noise caused by ftp1 being down for weeks. I confirmed the fix in rsync_files.sh is working. ftp2 has no logs since yesterday because it has been removed from the load balancer for testing purposes.