missing logs for releases.mozilla.com



mozilla.org Graveyard
Server Operations
5 years ago
3 years ago


(Reporter: aphadke, Assigned: ericz)



@wwwLocations = ("/mnt/data/stats/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp3.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/","/mnt/data/stats/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/");

No logs for ftp2.dmz.scl3.mozilla.com and ftp5.dmz.scl3.mozilla.com since 5/14.
Assignee: network-operations → server-ops
Component: Server Operations: Netops → Server Operations
QA Contact: ravi → shyam
Eric, I re-built and re-puppetized these machines, can you look into what was missed here and add it into puppet?
Assignee: server-ops → eziegenhorn

Comment 2

5 years ago
It's not compressing the logs, so they never get shipped.  The compression script appears to have been killed a year ago so I'm surprised we're just now running in to this.  Looking into integrating the compression into rsync_file.sh.

Comment 3

5 years ago
I noticed it while making changes for the CDH upgrade. I assume there's no way to get the data for earlier days? Can we have some checks in place to ensure this doesn't happen again?


Comment 4

5 years ago
Yes, the data from earlier days is still there and is on metrics-logger1 now.  We do have checks in place for this but this was lost in the noise caused by ftp1 being down for weeks.

I confirmed the fix in rsync_files.sh is working.  ftp2 has no logs since yesterday because it has been removed from the load balancer for testing purposes.
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.