1391041 - Fix logging for Symbols

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Comment 1

[Miles Crabill [:miles]]

Peter, it sounds like this might not be ready in prod for a little while. Should logging block taking Tecken to prod?

Comment 3

[Peter Bengtsson [:peterbe]]

(In reply to Miles Crabill [:miles] from comment #1)
> Peter, it sounds like this might not be ready in prod for a little while.
> Should logging block taking Tecken to prod?

I don't have a problem with it blocking the taking over of the symbols.mozilla.org domain. 

But, yes, I think this bug *should* block the "Going to prod with Tecken" *bug* and thus we shouldn't begin to try to use Tecken for things like making Tecken prod be a third URL for Socorro Processor (stackwalking) [0].

I am confident we can proceed with the domain name "takeover".


[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1383067

Comment 4

[Peter Bengtsson [:peterbe]]

Also, we need to document how other developers can do it too.

Comment 5

[Peter Bengtsson [:peterbe]]

Also, we need to remember that the logging for Celery goes elsewhere and devs need access to that too.

Comment 6

[Miles Crabill [:miles]]

Celery logging is accessible via the same means as tecken logging, and is working now in stage.

Comment 7

[Peter Bengtsson [:peterbe]]

Dev doesn't appear to be working. 

▶ ssh -A [e-mail] -t ssh symbols
Last login: Fri Aug 11 12:47:33 2017 from 172.31.51.111
This is instance i-0c36d1e45dd35f31a, a t2.micro in us-east-1d
Standard tags are: app => symbols, type => rawlogs, env => dev, stack => default
Public IP is 52.55.89.98 / ec2-52-55-89-98.compute-1.amazonaws.com

Welcome to the raw logs server. Logs for symbols may be found in /var/log/raw/

Older raw logs are available via: aws s3 ls s3://net-mozaws-dev-us-east-1-logging/symbols.

[pbengtsson@ip-172-31-45-204 ~]$ ls -ltr /var/log/raw/
total 0

Comment 8

[Peter Bengtsson [:peterbe]]

Also, we need to figure out a good place to document about this access. At the moment I have the instructions in my personal NOTES file which isn't very useful to others. 

I think we should add it to tecken.readthedocs.io somewhere but, Miles, it would be nice to have a blurb in there about what a dev needs to do to get the access sorted out for her account.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think the current convention for this sort of thing is:

1. Put architectural, method, function, getting started, and such on ReadTheDocs.

2. Put Mozilla-specific infrastructure, onboarding, how to (for things that require access to servers and talk about hosts), and such on Mana.

Miles should be writing infrastructure docs on Mana already for other ops people.

Logging is specifically Mozilla-specific infrastructure, so I'd put it there.

Comment 10

[Daniel Thorn [:relud]]

:peterbe I noticed from your linked output that dev is a t2.micro, which means :miles hasn't recycled it yet, so it would have issues reading kinesis. I've initiated recycling the host.

:willkg because logging is a common platform across all of cloudops' products, I'm the one who owns the docs for it. there are dev specific docs coming soon, and the current access doc is here: https://mana.mozilla.org/wiki/display/SVCOPS/Logging+2.0+Access+Retention+and+Delays

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

@daniel: That's awesome! But I think Peter is talking about logging 2.0 specifics for Tecken. That's what I'm talking about, too.

Just to clarify my poor wording, by "Logging is specifically Mozilla-specific infrastructure, so I'd put it there.", I really meant "Logging is specifically Mozilla-specific infrastructure, so I'd put logging-related access documentation for Tecken in Mana."

Comment 12

[Miles Crabill [:miles]]

I have created a Mana page describing the infrastructure for the Symbols service, including information about accessing logging.
See here: https://mana.mozilla.org/wiki/display/SVCOPS/Symbols.

Comment 13

[Miles Crabill [:miles]]

I have confirmed that logging is working for all envs. Future issues should be opened in another bug.