1176412 - Make Heroku report hosts separately to New Relic

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

Currently the Heroku New Relic python agents are reporting all of the dynos under "Dynamic Hostname" due to Heroku using dynamic host names, which is pretty unhelpful.

It looks like we can work around this using the latest version of the Python client (that we've not yet updated to), which adds a NEW_RELIC_PROCESS_HOST_DISPLAY_NAME env pref for manually specifying the host name that should appear in the APM UI:
https://docs.newrelic.com/docs/release-notes/agent-release-notes/python-release-notes/python-agent-252040

Comment 1

[Ed Morley [:emorley]]

I've submitted a support ticket to Heroku asking if setting the NEW_RELIC_PROCESS_HOST_DISPLAY_NAME env variable automatically to the name of the dyno would be something they could do (which would save us having to annotate the Procfile with export statements) - or if they have a better alternative.

Comment 2

[Ed Morley [:emorley]]

https://help.heroku.com/tickets/258508

Comment 3

[Ed Morley [:emorley]]

My original ticket:

> Our app reports to an existing New Relic account Mozilla have, rather than using
> the Heroku New Relic addon.
> 
> When viewing the app in the New Relic account, on the overview tab under the
> breakdown by server, all of the metrics are displayed under one entry called
> "Dynamic Hostname".
> 
> New versions of the New Relic Python client (plus Java and node.js clients)
> support setting the hostname either in newrelic.ini or in the environment as
> NEW_RELIC_PROCESS_HOST_DISPLAY_NAME (see https://docs.newrelic.com/docs/release-
> notes/agent-release-notes/python-release-notes/python-agent-252040).
> 
> I was wondering if Heroku could automatically set the environment variable
> NEW_RELIC_PROCESS_HOST_DISPLAY_NAME to be equal to the name of the dyno - which
> would presumably mean metrics would show up separately, and not all under
> "Dynamic Hostname"?
> 
> I know another solution would be to use the Heroku New Relic addon instead
> (which has the 'Instances' tab) - however since Mozilla already has a large
> allowance external New Relic account, it will be hard for me to justify the cost
> of using the addon just to get a couple of extra features.
> 
> Many thanks!

Their response:

> Hi Ed,
> 
> I don't think we'll be able to make specific changes like that to the overall
> environment, but you could probably adjust your Procfile line to export that
> environment variable to the value of the DYNO environment variable (which
> contains the dyno name). That would probably yield the result you're looking
> for. Will that work for you?
> 
> Thanks!

...so let's try using $DYNO :-)

Comment 4

[Ed Morley [:emorley]]

Thinking about this, it's more of a New Relic agent issue then a Heroku issue, so I've filed https://discuss.newrelic.com/t/avoiding-dynamic-hostname-on-heroku-using-the-name-provided-in-the-dyno-environment-variable/28430 against New Relic to see if they are more interested in fixing at the agent level.

Comment 5

[Ed Morley [:emorley]]

Attachment: Override the hostname reported by New Relic on Heroku

Comment 6

[Cameron Dawson [:camd]]

Comment on attachment 8653716 [details] [review]
Override the hostname reported by New Relic on Heroku

Very thorough detail!  :)

Comment 7

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/fce5c7e3c0a83dfac7412ca58028d3431f8ac831
Bug 1176412 - Override the hostname reported by New Relic on Heroku

On Heroku, the hostname is just a bunch of hex characters, which the New
Relic dashboard displays as "Dynamic Hostname". The New Relic Python
agent (as of v2.52.0.40) supports overriding this, by setting the
environment variable `NEW_RELIC_PROCESS_HOST_DISPLAY_NAME`:
https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#process_host-display_name

We use the value of `DYNO`, since this is set for us in each dyno's
environment, and is of form "web.2", "worker_pushlog.1" etc. Rather than
prefixing every line in the Procfile, we instead append the export to
the profile script created by the Python buildpack, so it's populated at
runtime on every dyno. See:
https://devcenter.heroku.com/articles/profiled

To do this we make use of the Python buildpack's `set-env` function:
https://github.com/heroku/heroku-buildpack-python/blob/ce3bdb37bafa3f56c54346d28ee5ef138cbc1893/bin/utils#L29-L32

Similar to what the buildpack does for its own env defines:
https://github.com/heroku/heroku-buildpack-python/blob/ce3bdb37bafa3f56c54346d28ee5ef138cbc1893/bin/compile#L190-L198

Comment 8

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #4)
> Thinking about this, it's more of a New Relic agent issue then a Heroku
> issue, so I've filed
> https://discuss.newrelic.com/t/avoiding-dynamic-hostname-on-heroku-using-the-
> name-provided-in-the-dyno-environment-variable/28430 against New Relic to
> see if they are more interested in fixing at the agent level.

Their reply:
https://discuss.newrelic.com/t/avoiding-dynamic-hostname-on-heroku-using-the-name-provided-in-the-dyno-environment-variable/28430

> Hi @emorley that's a great idea and I've gone ahead and filed a feature request on your behalf to have the agent check for the DYNO environment variable and use this to set the host display name. Thanks for your suggestion.

Though that doesn't block us, since we're setting it ourselves based on DYNO for now, as of the commit in comment 7 - it would just mean we can remove the workaround.

Comment 9

[Ed Morley [:emorley]]

Comment on attachment 8653716 [details] [review]
Override the hostname reported by New Relic on Heroku

So the PR here has been merged and is deployed on Heroku, however the New Relic UI doesn't appear to honour the host display name we provide in all places (it shows in some, so we know we're setting it ok).

I've posted in their APM Web UI forum section:
https://discuss.newrelic.com/t/apm-ui-doesnt-honour-custom-host-display-name-set-in-the-agent/28964

And will upgrade that to a support ticket if I don't hear back.

(In the meantime they've also made the post_compile set-env trick 'post of the week' hehe: https://discuss.newrelic.com/t/9-4-15-post-of-the-week-override-the-hostname-reported-by-new-relic-on-heroku/28920 )

Comment 10

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #9)
> I've posted in their APM Web UI forum section:
> https://discuss.newrelic.com/t/apm-ui-doesnt-honour-custom-host-display-name-
> set-in-the-agent/28964

Their response was:

> Hey @emorley - sorry for the delay in our response. We researched this with our developers and found that currently, the scope of the display host option doesn't include the additional items you mentioned, but we really appreciate your feedback about how those additions could be useful! We've passed your feedback on and will consider it for future expansions to this functionality.

Which is somewhat annoying.

I've replied; failing that I can fix this with a half-dozen line diff to the New Relic client, but we'd then be using our own fork.

Comment 11

[Ed Morley [:emorley]]

The librato graphs give detailed breakdowns, so I think we can live without this for the short term (which is fortunate, given the lack of a decent solution other than "fork the newrelic agent" or "wait for new relic to sort something").

Comment 12

[Ed Morley [:emorley]]

The latest version of the New Relic Python agent has just added support for this, making our lives a lot easier.

All we need to do is:
1) Update to the new version (https://github.com/mozilla/treeherder/pull/2843)
2) Remove the old workaround (to avoid possible weirdness in the NewRelic UI and to clean up)

Comment 13

[Ed Morley [:emorley]]

I've filed a New Relic Python agent ticket, asking them to add `release` to the list of dyno names to shorten (alongside `run` and `scheduler`):
https://support.newrelic.com/tickets/266737/edit

Comment 14

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:rm-newrelic-displayname > mozilla:master

Comment 15

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/08430370bd652446c5038fd098f63273cfccb92b
Bug 1176412 - Heroku: Remove New Relic host display name workaround (#2844)

Previously New Relic displayed the hostname for each dyno as
"Dynamic Hostname", due to the randomly generated Heroku hostnames.
As such, we had to set a manual display name to identify the dyno
type reported for each transaction. This was only partly useful, since
New Relic doesn't use it in all places of their UI.

However the New Relic Python agent 2.96.0.80 release has now added
official support for Heroku dyno names, making this workaround
unnecessary:
https://docs.newrelic.com/docs/release-notes/agent-release-notes/python-release-notes/python-agent-252040
https://github.com/edmorley/newrelic-python-agent/commit/d2c4aa097e18da56f6d2d21879796f7ef796f05a#diff-a57cabe85e94b1050fb4ff1749effcf5R335