prod fxa kibana filter beyond last 5 min breaks graphing

VERIFIED FIXED

Status

VERIFIED FIXED
5 years ago
5 years ago

People

(Reporter: edwong, Assigned: whd)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qa+])

(Reporter)

Description

5 years ago
1. sign in here:
https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json
2. in top right, change view to 1h

actual: it breaks the app, I can't filter and graph anymore.  Switching back to 5m doesn't fix things.
(Reporter)

Updated

5 years ago
Blocks: 972065
I am just going to take a guess that it is the very same symptom(s) as #32 above.
:-)
Whiteboard: [qa+]

Comment 3

5 years ago
fwiw, https://github.com/mozilla-services/puppet-config/issues/146 may help (this bug happens because elasticsearch falls over)
(Assignee)

Comment 4

5 years ago
I'm not seeing this on the new fxa-specific stack (yet, anyway):

https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json

We are currently not sending fxa app logs to this aggregator due to a bug in our heka 0.5.0 configuration:

https://github.com/mozilla-services/puppet-config/issues/180

The fact that we aren't may be related to it working better in the new environment, but I don't think we were indexing app logs in the first place (only nginx ones).
(Reporter)

Comment 5

5 years ago
Hmm - i can still repro this bug when using:
https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json
after i change the filter from 5m to 15m then back, i get no graphing data.
(Assignee)

Comment 6

5 years ago
Ok so something is definitely screwy here. I can now repro on the new instance. I will investigate persona/apache and other misc. settings. It's not related to load because we're still not sending app logs to this instance.
Assignee: nobody → whd
(Assignee)

Comment 7

5 years ago
So looks like this is related to a logstash-ism. Kibana is expecting the timestamp to be the "timestamp" field, when we're reporting it as "Timestamp". When you go to switch the default time it reverts from the latter to the former, causing nothing to be displayed. I need to figure out how to tweak the time widget so that it understands our timestamp format better.
(Reporter)

Comment 9

5 years ago
verified!
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
:whd before we close this out - have you made similar changes to the Stage version of this?
https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
(Assignee)

Comment 11

5 years ago
:jbonacci yes that kibana has the the change, however here is the link to the fxa-specific stage stack that we will use going forward:

https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json

This makes https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json obsolete¸ as fxa stage should be or will soon be pointing at the non-shared stack. I will document this on mana at some point, but see here for the issue on it: https://github.com/mozilla-services/puppet-config/issues/134

https://kibana.shared.us-east-1.stage.mozaws.net will house kibana dashboards for non-fxa cloud services, current examples being: https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync-Stage-Dash and https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/elasticsearch/APK%20Signer%20Log%20Search . Once things stop changing so much we'll get this all documented with proper kibana landing pages so you know which dashboards are where, and/or a meta-dashboard that points to the various others.
OK. Got it thanks.
Verified.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.