1483406 - New Relic Python agent "The generation of transaction data has failed"

Status: RESOLVED WORKSFORME

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

Spotted the following on Papertrail:

"""
Aug 14 07:10:00 treeherder-prod app/worker_default.2: [2018-08-14 06:10:00,045: WARNING/MainProcess] Soft time limit (600s) exceeded for fetch-bugs[1d56a718-f1a4-45bb-8a05-4e0f8479c942] 
Aug 14 07:10:00 treeherder-prod app/worker_default.2: 2018-08-14 06:10:00,046 (41/MainThread) newrelic.core.application ERROR - The generation of transaction data has failed. This would indicate some sort of internal implementation issue with the agent. Please report this problem to New Relic support for further investigation. 
Aug 14 07:10:00 treeherder-prod app/worker_default.2: Traceback (most recent call last): 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/application.py", line 842, in record_transaction 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     stats.record_transaction(data) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 887, in record_transaction 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     self.record_time_metrics(transaction.time_metrics(self)) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 520, in record_time_metrics 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     self.record_time_metric(metric) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 505, in record_time_metric 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     stats.merge_time_metric(metric) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 142, in merge_time_metric 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     self.merge_raw_time_metric(metric.duration, metric.exclusive) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 130, in merge_raw_time_metric 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     self[3] = self[0] and min(self[3], duration) or duration 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/billiard/pool.py", line 235, in soft_timeout_sighandler 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     raise SoftTimeLimitExceeded() 
Aug 14 07:10:00 treeherder-prod app/worker_default.2: SoftTimeLimitExceeded: SoftTimeLimitExceeded() 
Aug 14 07:10:00 treeherder-prod app/worker_default.2: [2018-08-14 06:10:00,046: ERROR/Worker-3] The generation of transaction data has failed. This would indicate some sort of internal implementation issue with the agent. Please report this problem to New Relic support for further investigation. 
Aug 14 07:10:00 treeherder-prod app/worker_default.2: Traceback (most recent call last): 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/application.py", line 842, in record_transaction 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     stats.record_transaction(data) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 887, in record_transaction 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     self.record_time_metrics(transaction.time_metrics(self)) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 520, in record_time_metrics 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     self.record_time_metric(metric) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 505, in record_time_metric 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     stats.merge_time_metric(metric) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 142, in merge_time_metric 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     self.merge_raw_time_metric(metric.duration, metric.exclusive) 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/core/stats_engine.py", line 130, in merge_raw_time_metric 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     self[3] = self[0] and min(self[3], duration) or duration 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/billiard/pool.py", line 235, in soft_timeout_sighandler 
Aug 14 07:10:00 treeherder-prod app/worker_default.2:     raise SoftTimeLimitExceeded() 
Aug 14 07:10:00 treeherder-prod app/worker_default.2: SoftTimeLimitExceeded: SoftTimeLimitExceeded() 
"""

It looks like the SoftTimeLimitExceeded was received whilst the New Relic agent was recording a transaction. Whilst this is a bit of an edge case, it's worth raising a ticket with New Relic as suggested in the error message above.

Comment 1

[Ed Morley [:emorley]]

This hasn't occurred since.