Closed Bug 1587952 Opened 5 years ago Closed 3 years ago

Add grafana trending for scriptworker-k8s queues

Categories

(Infrastructure & Operations :: RelOps: General, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Unassigned)

References

Details

The graphs for scriptworker-prov-v1 on the internal graphana are useful to see current and historical trends. It would be great to set that up for scriptworker-k8s too, with the set of workers at https://tools.taskcluster.net/provisioners/scriptworker-k8s/worker-types/.

Mihai, are we stable enough for the list of workers to go ahead ? IIRC it's a manually generated config. Or you might know of something we already have.

Flags: needinfo?(mtabara)

Oops, the supported graphs live on the other grafana.

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #1)

Oops, the supported graphs live on the other grafana.

Hm, two quick things before answering the question:

  1. TIL about these graphs exist, this is great!
  2. I can't login with the credentials in private repo. Have those been rotated or do I need to login via LDAP? I don't see Grafana in our sso.mozilla.com page so I've been slightly reluctant doing so.

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #0)

Mihai, are we stable enough for the list of workers to go ahead ? IIRC it's a manually generated config. Or you might know of something we already have.

Few days ago I would've said yes. But last week we also migrated all the workers (whether we're actually using them at any level as of now, or not) to mono-repo and we've also pushed to production for all of them. Some of them were failing last week (like snap), some are still not migrated and un-tested (like pushapk or treescript) and for some we're still waiting validation in beta1 this week, after mergeduty (like addonscript).

I'm leaning towards waiting until b1 is out (which validates a bunch of the workers at beta cycle level which is great) this week and also until we iron out the remaining issues with production push (I'm working on some of that today and tomorrow). So maybe towards the end of this week would be a good time to turn it on? What do you think?

Flags: needinfo?(mtabara)

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2)

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #1)

Oops, the supported graphs live on the other grafana.

Hm, two quick things before answering the question:

  1. TIL about these graphs exist, this is great!
  2. I can't login with the credentials in private repo. Have those been rotated or do I need to login via LDAP? I don't see Grafana in our sso.mozilla.com page so I've been slightly reluctant doing so.

Ah, forgot to drop a NI for this in my previous comment.

Flags: needinfo?(nthomas)

I have been using SSO but hadn't noticed that it wasn't on the dashboard. You could contact bpitts about that.

Good call on waiting to see how b1 goes.

Flags: needinfo?(nthomas)

Seems like beta1 went smooth! \o/

Lets go ahead here then. Is there anything we can do to help ?

Flags: needinfo?(dhouse)

https://earthangel-b40313e5.influxcloud.net/d/-k_LUaoZz/provisioner-queues-copy?orgId=1&refresh=5m&var-provisioner=scriptworker-prov-v1&var-provisioner=scriptworker-k8s&var-workerType=All&var-filter=workerType%7C%3D~%7C%2Fsigning%2F

I added the scriptworker-k8s provisioner to the collection and query in this copy of the scriptworker dashboard. We can replace/update the dashboard with this if it looks correct.

Flags: needinfo?(dhouse)

Thanks, all the workers seem to be there, and a couple of non-zero values make sense.

Unrelated, the tooltips are very tall and sometimes the useful values are drawn off the top of the screen. Does grafana have a way to only show non-zero values on the tooltip ?

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #8)

Thanks, all the workers seem to be there, and a couple of non-zero values make sense.

Unrelated, the tooltips are very tall and sometimes the useful values are drawn off the top of the screen. Does grafana have a way to only show non-zero values on the tooltip ?

There's an issue reported for that:
https://github.com/grafana/grafana/issues/1381

The partial fix (I just turned on) hides series with zero/null across the full time window. But there is still a bug/problem with showing localized zeros in the tooltip.

It can alternatively be changed to only show the series value under the cursor (single-value).

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

The tooltip looks great now, thanks. I think you were going to replace/update the original dashboard location (comment #2) with the new provisioner ?

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 5 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.