Closed Bug 1483406 Opened 7 years ago Closed 6 years ago

New Relic Python agent "The generation of transaction data has failed"

Categories

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Product:
Tree Management
Component:
Treeherder: Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: emorley, Assigned: emorley)

References

Details

Spotted the following on Papertrail: """ Aug 14 07:10:00 treeherder-prod app/worker_default.2: [2018-08-14 06:10:00,045: WARNING/MainProcess] Soft time limit (600s) exceeded for fetch-bugs[1d56a718-f1a4-45bb-8a05-4e0f8479c942] Aug 14 07:10:00 treeherder-prod app/worker_default.2: 2018-08-14 06:10:00,046 (41/MainThread) newrelic.core.application ERROR - The generation of transaction data has failed. This would indicate some sort of internal implementation issue with the agent. Please report this problem to New Relic support for further investigation. Aug 14 07:10:00 treeherder-prod app/worker_default.2: Traceback (most recent call last): Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/application.py", line 842, in record_transaction Aug 14 07:10:00 treeherder-prod app/worker_default.2: stats.record_transaction(data) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 887, in record_transaction Aug 14 07:10:00 treeherder-prod app/worker_default.2: self.record_time_metrics(transaction.time_metrics(self)) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 520, in record_time_metrics Aug 14 07:10:00 treeherder-prod app/worker_default.2: self.record_time_metric(metric) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 505, in record_time_metric Aug 14 07:10:00 treeherder-prod app/worker_default.2: stats.merge_time_metric(metric) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 142, in merge_time_metric Aug 14 07:10:00 treeherder-prod app/worker_default.2: self.merge_raw_time_metric(metric.duration, metric.exclusive) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 130, in merge_raw_time_metric Aug 14 07:10:00 treeherder-prod app/worker_default.2: self[3] = self[0] and min(self[3], duration) or duration Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/billiard/pool.py", line 235, in soft_timeout_sighandler Aug 14 07:10:00 treeherder-prod app/worker_default.2: raise SoftTimeLimitExceeded() Aug 14 07:10:00 treeherder-prod app/worker_default.2: SoftTimeLimitExceeded: SoftTimeLimitExceeded() Aug 14 07:10:00 treeherder-prod app/worker_default.2: [2018-08-14 06:10:00,046: ERROR/Worker-3] The generation of transaction data has failed. This would indicate some sort of internal implementation issue with the agent. Please report this problem to New Relic support for further investigation. Aug 14 07:10:00 treeherder-prod app/worker_default.2: Traceback (most recent call last): Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/application.py", line 842, in record_transaction Aug 14 07:10:00 treeherder-prod app/worker_default.2: stats.record_transaction(data) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 887, in record_transaction Aug 14 07:10:00 treeherder-prod app/worker_default.2: self.record_time_metrics(transaction.time_metrics(self)) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 520, in record_time_metrics Aug 14 07:10:00 treeherder-prod app/worker_default.2: self.record_time_metric(metric) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 505, in record_time_metric Aug 14 07:10:00 treeherder-prod app/worker_default.2: stats.merge_time_metric(metric) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 142, in merge_time_metric Aug 14 07:10:00 treeherder-prod app/worker_default.2: self.merge_raw_time_metric(metric.duration, metric.exclusive) Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 130, in merge_raw_time_metric Aug 14 07:10:00 treeherder-prod app/worker_default.2: self[3] = self[0] and min(self[3], duration) or duration Aug 14 07:10:00 treeherder-prod app/worker_default.2: File "/app/.heroku/python/lib/python2.7/site-packages/billiard/pool.py", line 235, in soft_timeout_sighandler Aug 14 07:10:00 treeherder-prod app/worker_default.2: raise SoftTimeLimitExceeded() Aug 14 07:10:00 treeherder-prod app/worker_default.2: SoftTimeLimitExceeded: SoftTimeLimitExceeded() """ It looks like the SoftTimeLimitExceeded was received whilst the New Relic agent was recording a transaction. Whilst this is a bit of an edge case, it's worth raising a ticket with New Relic as suggested in the error message above.
This hasn't occurred since.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.