Add grafana trending for scriptworker-k8s queues
Categories
(Infrastructure & Operations :: RelOps: General, enhancement)
Tracking
(Not tracked)
People
(Reporter: nthomas, Unassigned)
References
Details
The graphs for scriptworker-prov-v1 on the internal graphana are useful to see current and historical trends. It would be great to set that up for scriptworker-k8s too, with the set of workers at https://tools.taskcluster.net/provisioners/scriptworker-k8s/worker-types/.
Mihai, are we stable enough for the list of workers to go ahead ? IIRC it's a manually generated config. Or you might know of something we already have.
|
Reporter | |
Comment 1•5 years ago
|
||
Oops, the supported graphs live on the other grafana.
|
||
Comment 2•5 years ago
•
|
||
(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #1)
Oops, the supported graphs live on the other grafana.
Hm, two quick things before answering the question:
- TIL about these graphs exist, this is great!
- I can't login with the credentials in private repo. Have those been rotated or do I need to login via LDAP? I don't see Grafana in our sso.mozilla.com page so I've been slightly reluctant doing so.
(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #0)
Mihai, are we stable enough for the list of workers to go ahead ? IIRC it's a manually generated config. Or you might know of something we already have.
Few days ago I would've said yes. But last week we also migrated all the workers (whether we're actually using them at any level as of now, or not) to mono-repo and we've also pushed to production for all of them. Some of them were failing last week (like snap), some are still not migrated and un-tested (like pushapk or treescript) and for some we're still waiting validation in beta1 this week, after mergeduty (like addonscript).
I'm leaning towards waiting until b1 is out (which validates a bunch of the workers at beta cycle level which is great) this week and also until we iron out the remaining issues with production push (I'm working on some of that today and tomorrow). So maybe towards the end of this week would be a good time to turn it on? What do you think?
|
|
||
Comment 3•5 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2)
(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #1)
Oops, the supported graphs live on the other grafana.
Hm, two quick things before answering the question:
- TIL about these graphs exist, this is great!
- I can't login with the credentials in private repo. Have those been rotated or do I need to login via LDAP? I don't see Grafana in our sso.mozilla.com page so I've been slightly reluctant doing so.
Ah, forgot to drop a NI for this in my previous comment.
|
|
Reporter | |
Comment 4•5 years ago
|
||
I have been using SSO but hadn't noticed that it wasn't on the dashboard. You could contact bpitts about that.
Good call on waiting to see how b1 goes.
|
|
||
Comment 5•5 years ago
|
||
Seems like beta1 went smooth! \o/
|
|
Reporter | |
Comment 6•5 years ago
|
||
Lets go ahead here then. Is there anything we can do to help ?
|
|
Reporter | |
Updated•5 years ago
|
I added the scriptworker-k8s provisioner to the collection and query in this copy of the scriptworker dashboard. We can replace/update the dashboard with this if it looks correct.
|
|
Reporter | |
Comment 8•5 years ago
•
|
||
Thanks, all the workers seem to be there, and a couple of non-zero values make sense.
Unrelated, the tooltips are very tall and sometimes the useful values are drawn off the top of the screen. Does grafana have a way to only show non-zero values on the tooltip ?
(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #8)
Thanks, all the workers seem to be there, and a couple of non-zero values make sense.
Unrelated, the tooltips are very tall and sometimes the useful values are drawn off the top of the screen. Does grafana have a way to only show non-zero values on the tooltip ?
There's an issue reported for that:
https://github.com/grafana/grafana/issues/1381
The partial fix (I just turned on) hides series with zero/null across the full time window. But there is still a bug/problem with showing localized zeros in the tooltip.
It can alternatively be changed to only show the series value under the cursor (single-value).
|
|
Reporter | |
Comment 10•5 years ago
|
||
The tooltip looks great now, thanks. I think you were going to replace/update the original dashboard location (comment #2) with the new provisioner ?
Description
•