1082771 - Add a static "Infrastructure" menu that links to existing dashboards

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: infrastructure.png

TBPL has a menu that lists the total counts of pending/running jobs on all trees, broken down by tree, along with links to the following pages:
http://builddata.pub.build.mozilla.org/reports/pending/pending.html
https://secure.pub.build.mozilla.org/buildapi/running
http://builddata.pub.build.mozilla.org/reports/pending/running.html
https://secure.pub.build.mozilla.org/buildapi/pending


Would be nice to have access to this information from within Treeherder.

Comment 1

[Ed Morley [:emorley]]

We could also add a few more links to the menu - eg some of the examples given in bug 1124059.

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I'd argue this should be higher priority as we approach end of life for TBPL.

Comment 3

[Ed Morley [:emorley]]

So the TBPL implementation works like this:
* Load builds-{pending,running}.js client side
* In addition to overlaying pending/running jobs on the currently visible pushes, count _all_ pending/running jobs from all repos & use those figures in the infra menu

Now with Treeherder:
1) We don't load builds-{pending,running}.js in the client, it's server side.
2) The picture is a bit more complicated, since builds-{pending,running}.js only covers buildbot and not taskcluster. Plus how do we want to break this down? Should taskcluster even be lumped into the same display?
3) In order to get a total overview of pending/running jobs across all repos, we'd have to query many many DBs, since each repo has it's own DB - plus we'd have to query back N days/weeks, since there is no guarantee that the pending/running job is recent. Though I guess we could use the last_modified field to limit the query to as few rows as possible, to keep it performant.

Plus for a true picture of the "current state of the infra", I think simple pending/running counts are no longer really that helpful. You really need:
* Pending/running counts broken down by machine type and repo
* Number of available machines of each type
* Number of machines that have not completed a job in the last N minutes
* AWS info (jacuzzi status/thresholds etc)
* Age of the oldest job in the pending queue
* Graphs to show trends over the last N hours, plus to compare to the same day last week etc
etc 

I'm thinking that some of the above is maybe better served upstream (in taskcluster/buildbot) than by Treeherder. And even if we want to pull all that together in Treeherder, we'll still need better API support/datasources (or at least to figure existing ones out, if they are already in place) from upstream first.

As such (and so we can get at least something to replace the infra menu for now), I think the first iteration should just be a bunch of links to buildbot/treeherder pages, with no live stats displayed within Treeherder. For me at least, the buildbot graphs provided much more value than the raw stats, such that I often jumped straight to those pages even when I was using TBPL - so hopefully this shouldn't be too much of an issue.

Things we could link to:

https://secure.pub.build.mozilla.org/buildapi/pending
http://builddata.pub.build.mozilla.org/reports/pending/pending.html
https://secure.pub.build.mozilla.org/buildapi/running
http://builddata.pub.build.mozilla.org/reports/pending/running.html
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/

Already linked from slave_health (but would probably be good to link to directly):
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/build_trends.html
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/try_trends.html
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/test_trends.html
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/jacuzzis.html
hg status: http://nigelbabu.github.io/hgstats/#hours/2
EC2 dashboard: https://www.hostedgraphite.com/da5c920d/grafana/#/dashboard/temp/5d509a54d0b496d184023bab35c17d0f41e6c607
Runner dashboard: https://stats.taskcluster.net/grafana/#/dashboard/db/runner

We could also move things like the clobberer link from the tree info menu to infrastructure.

@sheriffs: What are your preferences for the above? Which are the most useful? Any other links we should include?

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

That would be sufficient for me, I think, at least in the short term.

Comment 5

[Ed Morley [:emorley]]

@sherrifs: Which of the links to dashboards in comment 3 are useful to you? Are there any other things I should be linking to? Perhaps we could make this a general "Status" menu (ie status of not just the build infra, but say Treeherder New Relic too?)

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I would say the following would be useful in there:

https://secure.pub.build.mozilla.org/buildapi/pending
http://builddata.pub.build.mozilla.org/reports/pending/pending.html
https://secure.pub.build.mozilla.org/buildapi/running
http://builddata.pub.build.mozilla.org/reports/pending/running.html
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/

Clobberer link could probably be moved to Infrastructure (the tree-specific links from the tree-info links don't actually seem to get the clobberer page to pre-select the specified tree, so just a generic link to the clobberer under Infrastructure would be enough).

I don't really use anything beyond those, but I generally redirect to Releng if those don't let me figure out what's going on, so I don't know if the other links would be useful in there.

Comment 7

[Ryan VanderMeulen (PTO, back July 27)]

Wes' list looks good. An EC2 dashboard link would be nice as well.

Comment 8

[Ed Morley [:emorley]]

Attachment: Add an infra menu that links to dashboards/tools

Comment 9

[Jonathan French (:jfrench)]

Comment on attachment 8580801 [details] [review]
Add an infra menu that links to dashboards/tools

Lgtm, and any minor cosmetic tweaking we can do later :) So long this has been tested locally and it's behaving properly, all good on my end.

Comment 10

[Jonathan French (:jfrench)]

Actually I just tried it locally also, and it's cool. The only thing I can note is some of the menu entries require LDAP/Level3 (I think):

BuildAPI Pending
BuildAPI Running
Slave Health (kind of..)
Clobberer

versus the other Infra menu entries which don't. Perhaps there is visual differentiation we could provide in all menus (Infra, [i], resultset bar menu) to cue the user menus which are not accessible without the appropriate level-access (eg. disabled styling?, not sure). It's outside the scope of the fix here though.

The only other fix observations are:

1) tree status(per repo) and clobberer(per repo) already in the little "[i]" icon adjacent to the current repo in the navbar, so I assume we are ok for having somewhat similar (but yes, differently addressed) menus in two places.

2) the case treatment and word-space style in the repo [i] menu is different, we could unify their appearance later in another bug.

3) the new Infra menu itself in the navbar is available when not logged in to the platform, just to confirm that is the intent? (makes sense to me, even with above Level-access issues).

I stills stand by my r+, in any case :)

Comment 11

[Ed Morley [:emorley]]

(In reply to Jonathan French (:jfrench) from comment #10)
> Actually I just tried it locally also, and it's cool. The only thing I can
> note is some of the menu entries require LDAP/Level3 (I think):

Yeah this is unavoidable, and was the case with TBPL too.

> versus the other Infra menu entries which don't. Perhaps there is visual
> differentiation we could provide in all menus 

Good idea - let's do this in a followup :-)

> 1) tree status(per repo) and clobberer(per repo) already in the little "[i]"
> icon adjacent to the current repo in the navbar, so I assume we are ok for
> having somewhat similar (but yes, differently addressed) menus in two places.

Yeah the idea was that we can make the repo menu links be deep links and the ones on the infra menu be to the overview pages. TBH I think the best plan is just to add them to the infra menu for now and see over time which people prefer to use, and consider removing the other if appropriate.

> 2) the case treatment and word-space style in the repo [i] menu is
> different, we could unify their appearance later in another bug.

Ah another good spot :-) Let's sort this with the other followup above.

> 3) the new Infra menu itself in the navbar is available when not logged in
> to the platform, just to confirm that is the intent? (makes sense to me,
> even with above Level-access issues).

Yeah this is intended - none of the links are any more accessible when people are logged into persona.

Thank you for the review and testing! :-D

Comment 12

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-ui

https://github.com/mozilla/treeherder-ui/commit/021b986a84077e6b9d8545e64e54a9d965282e37
Bug 1082771 - Close RepositoryMenuCtrl span sooner

https://github.com/mozilla/treeherder-ui/commit/02e588f5e1d6fac926daf88f9256e0b15b6bb149
Bug 1082771 - Clean up comments in thGlobalTopNavPanel.html

https://github.com/mozilla/treeherder-ui/commit/d137202b2cff40d88669f16c876293a47cadadf2
Bug 1082771 - Add an infra menu that links to dashboards/tools

Comment 13

[Jonathan French (:jfrench)]

(In reply to Ed Morley (Away 23rd March -> 1st April) [:edmorley] from comment #11)
> 
> Ah another good spot :-) Let's sort this with the other followup above.

Entered as bug 1145837 :)

Comment 14

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-ui

https://github.com/mozilla/treeherder-ui/commit/6ebc35060f8f6ae9eb856d32441906feaaf15989
Bug 1082771 - Add headings/separators to the infra menu

Comment 15

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/5e19ca8fe6c64226a109fe4003df6d95edb6208b
Bug 1082771 - Close RepositoryMenuCtrl span sooner

https://github.com/mozilla/treeherder/commit/aadb15239742a2a546c426aa91706f61b367d64e
Bug 1082771 - Clean up comments in thGlobalTopNavPanel.html

https://github.com/mozilla/treeherder/commit/2075df97ca65b5e42765bd3634011d7efa8704fe
Bug 1082771 - Add an infra menu that links to dashboards/tools

https://github.com/mozilla/treeherder/commit/b2771477632a053be2cc972ce7e00acafdca7824
Bug 1082771 - Add headings/separators to the infra menu