No log files showing up from pp-zlb01.phx.mozilla.net after 2011-03-22-14

RESOLVED FIXED

Status

--
critical
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: nelson.sousa, Assigned: cshields)

Tracking

Details

(Reporter)

Description

8 years ago
We parse crash stats from the crash-report log files mounted on  /mnt/stats_im-log03/stats/logs/pp-zlb0[15].phx.mozilla.net.
There are no new files on pp-zlb01 since 2011-03-22-14. These files are parsed from our etl to generate crash statistics to be included on the nightly report and several dashboards.
(Assignee)

Updated

8 years ago
Assignee: server-ops → cshields
Did we have a dip in traffic as well? It is possible the server is mot actively serving data for that pool anymore.
Are there files for other sites?

Also, you should be parsing data from pp-zlb06 as well.
(Assignee)

Comment 2

8 years ago
someone had added im-log03 to the same line as im-log02 in /etc/hosts on pp-zlb01 on that day, which redirected the im-log03 copies.

I've fixed that and am in the process of copying the files back
(Reporter)

Comment 3

8 years ago
Files are visible now. Thanks for the quick response time.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
(Assignee)

Comment 4

8 years ago
not complete yet, I'm still copying the missing files.
(Reporter)

Comment 5

8 years ago
my fault. Looked to the filelist, saw the 2011-03-24-04 file listed last, didn't notice there was still a gap in the middle. Thanks
(Assignee)

Comment 6

8 years ago
should be fixed and everything missing replaced.  Please verify
Okay, that is a relief because it also explains why none of the other log processing systems were affected. They automatically fall back to im-log02 if expected files arent found on 03.
(Assignee)

Comment 8

8 years ago
(In reply to comment #7)
> Okay, that is a relief because it also explains why none of the other log
> processing systems were affected. They automatically fall back to im-log02 if
> expected files arent found on 03.

yeah, and this was a manual change.  I've sent email out, will investigate why this was done in the first place.
Corey,

This was a feature/bug in the way puppet updated /etc/hosts. I'll go into details with you offline.
(Reporter)

Comment 10

8 years ago
all files accounted for. thanks.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.