878172 - missing logs for releases.mozilla.com

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Anurag Phadke[:[e-mail]]]

@wwwLocations = ("/mnt/data/stats/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp3.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/");

No logs for ftp2.dmz.scl3.mozilla.com and ftp5.dmz.scl3.mozilla.com since 5/14.

Comment 1

[Shyam Mani [:fox2mike]]

Eric, I re-built and re-puppetized these machines, can you look into what was missed here and add it into puppet?

Comment 2

[Eric Ziegenhorn :ericz]

It's not compressing the logs, so they never get shipped.  The compression script appears to have been killed a year ago so I'm surprised we're just now running in to this.  Looking into integrating the compression into rsync_file.sh.

Comment 3

[Anurag Phadke[:[e-mail]]]

I noticed it while making changes for the CDH upgrade. I assume there's no way to get the data for earlier days? Can we have some checks in place to ensure this doesn't happen again?

-anurag

Comment 4

[Eric Ziegenhorn :ericz]

Yes, the data from earlier days is still there and is on metrics-logger1 now.  We do have checks in place for this but this was lost in the noise caused by ftp1 being down for weeks.

I confirmed the fix in rsync_files.sh is working.  ftp2 has no logs since yesterday because it has been removed from the load balancer for testing purposes.