1124278 - Improve annotations shown in new relic for common errors

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

"New Relic allows you to collect standard attributes, as well as custom attributes, to several destinations. For example, you may want to track the user name associated with a slow or failing request."

It seems that for some transactions we're not even getting the basic annotations, and for others we could improve what's reported, using:
https://docs.newrelic.com/docs/apm/other-features/attributes/collecting-custom-attributes

This would help with things like bug 1124270, since at present we don't even know what log or job type was being parsed. Same for pushlog errors - we have the repo but not the URL requested etc.

Plus for Bugzilla API comment submission (bug 1114785).

Comment 1

[Ed Morley [:emorley]]

This has caused me many hours of busywork over the months and today was the last straw, let's fix it.

Comment 2

[Ed Morley [:emorley]]

Note to self: Also see if we can improve the Celery retry exceptions, so they don't get lumped together (a la the ones listed in bug 1152896 comment 1).

Comment 3

[Ed Morley [:emorley]]

I'm going to punt this until after Heroku, since it will need a bit of experimentation to figure out the right place to annotate the tasks & what level of detail is appropriate, plus we'll need to wrap newrelic.agent.add_custom_parameter, to handle non-stage/prod cases where we aren't using new relic.

Comment 4

[Ed Morley [:emorley]]

We can also include the hawk client id in the list of attributes too :-)

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/c2d1100bd56b65959e1ca44525e2bd08b141ea94
Bug 1124278 - Include hawk user id in hawk calls

This will add that value to any exceptions that are caused during calls by a hawk
user.  This will, in turn, help us work with that user to resolve the issue.

Comment 6

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:newrelic-custom-attributes > mozilla:master

Comment 7

[William Lachance (:wlach)]

Comment on attachment 8758164 [details] [review]
[treeherder] mozilla:newrelic-custom-attributes > mozilla:master

LGTM!

Comment 8

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/8acd401cb3dfd6f5d6f22b0a73ac2269d7fa7903
Bug 1124278 - Add New Relic custom attributes for more celery tasks

Requests made to the Treeherder API have their URLs and other request
metadata recorded by the New Relic agent, which helps when debugging.

However background celery tasks have no such URL or request metadata,
so really benefit from the addition of custom attributes. These are then
shown in the New Relic errors dashboard, slow transaction traces and
in the Insights analysis tool:
https://docs.newrelic.com/docs/agents/manage-apm-agents/agent-metrics/collect-custom-attributes
https://docs.newrelic.com/docs/agents/python-agent/customization-extension/python-transaction-api#add_custom_parameter

Comment 9

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:newrelic-attributes-tweaks > mozilla:master

Comment 10

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/6bcbab885cbb929e965df646ca038b9050d72f60
Bug 1124278 - Adjust the New Relic custom attributes for the log parser (#1583)

* Adds name and url for each JobLog, to save having to look them up.
* Renames the status attribute to make it clearer that it refers to the
status that was set prior to the task starting (eg helps answer if the
log was already successfully parsed or had previously failed).
* Switches the status to the human-readable status descriptions (such as
"pending") rather than the status code, using `Model.get_FOO_display()`:
https://docs.djangoproject.com/en/1.8/ref/models/instances/#django.db.models.Model.get%5FFOO%5Fdisplay