1091744 - No hg.m.o logs after 2014-10-15

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[hwine]

The zeus access logs stop on Oct 15:

[[e-mail]3 logs]$ find . -type f -name hg.mozilla.org.access_2014-10-15\* | wc -l
48
[[e-mail]3 logs]$ find . -type f -name hg.mozilla.org.access_2014-10-16\* | wc -l
0
[[e-mail]3 logs]$ pwd
/data/logs

Comment 1

[C. Liang [:cyliang]]

I've re-established the collection of access logs for hg.mozilla.org.  These *should* start landing on metrics-logger1 within the next two hours. 

Back around the 15th of October, Metrics gave WebOps a list of the log files on metrics-logger1 that they actually process.  We stopped shipping log files to metrics-logger1 for anything not on that list.  This brings me to two questions:

   1) Are there any other log files that you would like to see that aren't being updated?  For example, I can see that we're not shipping logs for reviewboard.mozilla.org, git.mozilla.org, etc.

   2) Is there a better way / place to ship these files to you?

Comment 2

[hwine]

I'll defer to fubar on this.

Comment 3

[Kendall Libby [:fubar] (he/him)]

(In reply to C. Liang [:cyliang] from comment #1)
> Back around the 15th of October, Metrics gave WebOps a list of the log files
> on metrics-logger1 that they actually process.  We stopped shipping log
> files to metrics-logger1 for anything not on that list.  This brings me to
> two questions:

That might have been a nice thing to announce to infra-all. AFAIK, that was the only location where logs have been aggregated from Zeus. Also, changing the location where Zeus itself logs has caused a misstep in trying to diagnose problems
(I know it's affected us and MOC, presumably others). 


>    1) Are there any other log files that you would like to see that aren't
> being updated?  For example, I can see that we're not shipping logs for
> reviewboard.mozilla.org, git.mozilla.org, etc.

All Dev Services sites: MXR, DXR, reviewboard, treeherder, git, hg, svn, cvs, bzr, bugzilla, (-dev)?.(mozilla|allizom).org. 

>    2) Is there a better way / place to ship these files to you?

We're looking at building our own logstash instance some time in 2015, but until then not really. Syslog to syslog1.private might be an option, but the config is awful to parse, I'm not sure if the host is up for it, and BMO already logs from the web heads there so I'm concerned about overlap.

Comment 4

[C. Liang [:cyliang]]

I made a first pass at trying to restore logging functionality but have some properties that will require  me to confer with fubar.

Comment 5

[C. Liang [:cyliang]]

Attachment: devsvcs_logs.txt

I've attached a list of virtual server -> log file name mappings.  Most log files should already be find-able on metrics-logger; dxr.mozilla.org logs should land there starting today.