1. sign in here: https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json 2. in top right, change view to 1h actual: it breaks the app, I can't filter and graph anymore. Switching back to 5m doesn't fix things.
I am just going to take a guess that it is the very same symptom(s) as #32 above. :-)
fwiw, https://github.com/mozilla-services/puppet-config/issues/146 may help (this bug happens because elasticsearch falls over)
I'm not seeing this on the new fxa-specific stack (yet, anyway): https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json We are currently not sending fxa app logs to this aggregator due to a bug in our heka 0.5.0 configuration: https://github.com/mozilla-services/puppet-config/issues/180 The fact that we aren't may be related to it working better in the new environment, but I don't think we were indexing app logs in the first place (only nginx ones).
Hmm - i can still repro this bug when using: https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json after i change the filter from 5m to 15m then back, i get no graphing data.
Ok so something is definitely screwy here. I can now repro on the new instance. I will investigate persona/apache and other misc. settings. It's not related to load because we're still not sending app logs to this instance.
Assignee: nobody → whd
So looks like this is related to a logstash-ism. Kibana is expecting the timestamp to be the "timestamp" field, when we're reporting it as "Timestamp". When you go to switch the default time it reverts from the latter to the former, causing nothing to be displayed. I need to figure out how to tweak the time widget so that it understands our timestamp format better.
I think this did the trick: https://github.com/mozilla-services/puppet-config/commit/0c1c32dd5846f085f1f6b050fb95a5b45c1ffb92 :edwong or :jbonacci can you verify https://kibana.fxa.us-west-2.prod.mozaws.net/index.html#/dashboard/file/weblogs.json works now?
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
:whd before we close this out - have you made similar changes to the Stage version of this? https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json
:jbonacci yes that kibana has the the change, however here is the link to the fxa-specific stage stack that we will use going forward: https://kibana.fxa.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json This makes https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/weblogs.json obsolete¸ as fxa stage should be or will soon be pointing at the non-shared stack. I will document this on mana at some point, but see here for the issue on it: https://github.com/mozilla-services/puppet-config/issues/134 https://kibana.shared.us-east-1.stage.mozaws.net will house kibana dashboards for non-fxa cloud services, current examples being: https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/elasticsearch/Sync-Stage-Dash and https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/elasticsearch/APK%20Signer%20Log%20Search . Once things stop changing so much we'll get this all documented with proper kibana landing pages so you know which dashboards are where, and/or a meta-dashboard that points to the various others.
OK. Got it thanks. Verified.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.