1114785 - TBPLbot fails to comment in bugs when the failure lines make the comment exceed 65535 characters

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Ryan VanderMeulen [:RyanVM]]

I'm assuming this is fallout from today's changes. I just classified the Linux64 PGO bc1 job in the below push as bug 1090633, but the comment never appeared in the bug. Maybe relevant is that the bug is RESOLVED FIXED and I had to click on Show/Hide More to see it as an option.
https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-beta&revision=a476f2afa8f9

Comment 1

[Ryan VanderMeulen [:RyanVM]]

That said, I've seen other bugs pick up comments, so there must be something special about this case.

Comment 2

[Ed Morley [:emorley]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #0)
> I'm assuming this is fallout from today's changes.

Nothing has landed yet wrt bug 1069560 and friends, so I don't think this is related to a code change (at least not in the service, unless one of the UI changes for the classification workflow UX broke something).

Comment 3

[Ryan VanderMeulen [:RyanVM]]

I was thinking maybe related to the TBPL changes that landed today.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

It's also entirely possible that this was always broken and just never noticed before :)

Comment 5

[Ed Morley [:emorley]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #3)
> I was thinking maybe related to the TBPL changes that landed today.

Bug 1114581 only modified starcomment.php which is a standalone file just for ElasticSearch submission.

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #4)
> It's also entirely possible that this was always broken and just never
> noticed before :)

Yeah true :-)
It may be bug 887313 or something similar striking again.
At least once we have bug 1069560 we'll have one less step of indirection and can log the error from the Bugzilla REST API directly if needs be :-)

Comment 6

[Ed Morley [:emorley]]

I've looked on https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors and can't see anything related to classification sadly.

Comment 7

[Ed Morley [:emorley]]

Now that we're submitting to Bugzilla directly, we can see any errors coming from Bugzilla. Looking at:
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors

...and adjusting the timeframe to longer than the last 30 minutes, plus filtering by "submit-bug-comment", gives:

celery.exceptions:Retry: Retry in 180s: HTTPError('400 Client Error: Bad Request',) 

Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/newrelic-2.40.0.34/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs)
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tasks/tbpl_tasks.py", line 40, in submit_bug_comment submit_bug_comment.retry(exc=e)
File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 680, in retry raise ret
Retry: Retry in 180s: HTTPError('400 Client Error: Bad Request',)

requests.exceptions:HTTPError: 400 Client Error: Bad Request 

Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/newrelic-2.40.0.34/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs)
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tasks/tbpl_tasks.py", line 40, in submit_bug_comment submit_bug_comment.retry(exc=e)
File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 660, in retry maybe_reraise()
File "/usr/lib/python2.6/site-packages/celery/utils/__init__.py", line 242, in maybe_reraise reraise(exc_info[0], exc_info[1], exc_info[2])
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tasks/tbpl_tasks.py", line 38, in submit_bug_comment req.send_request()
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tbpl.py", line 163, in send_request r.raise_for_status()
File "/usr/lib/python2.6/site-packages/requests/models.py", line 773, in raise_for_status raise HTTPError(http_error_msg, response=self)
HTTPError: 400 Client Error: Bad Request


I've found one instance in the celery log on rabbitmq1 - looks like the comment was just too large (it's pages and pages):
[2015-01-13 15:56:14,628: INFO/Worker-6] Sending data to https://bugzilla.mozilla.org/rest/bug/1114541/comment: {'comment': u'submit_timestamp: 2015-01-13T15:26:04\nlog: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=5420255\nrepository: mozilla-inbound\nwho: wkocher[at]mozilla[dot]com\nmachine: tst-linux64-spot-967\nbuildname: Ubuntu VM 12.04 x64 mozilla-inbound opt test mochitest-browser-chrome-3\nrevision: 544315e9741c\n\n
...
PROCESS-CRASH | toolkit/mozapps/extensions/test/xpinstall/browser_bug540558.js | application crashed [@ libc-2.15.so + 0xe8403]\nReturn code: 1'}
[2015-01-13 15:56:14,630: INFO/Worker-6] Starting new HTTPS connection (1): bugzilla.mozilla.org
[2015-01-13 15:56:15,011: ERROR/MainProcess] Task submit-bug-comment[c3319a20-0ef8-4580-9656-6ee555d1b855] raised unexpected: HTTPError('400 Client Error: Bad Request',)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/newrelic-2.40.0.34/newrelic/hooks/application_celery.py", line 66, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tasks/tbpl_tasks.py", line 40, in submit_bug_comment
    submit_bug_comment.retry(exc=e)
  File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 660, in retry
    maybe_reraise()
  File "/usr/lib/python2.6/site-packages/celery/utils/__init__.py", line 242, in maybe_reraise
    reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tasks/tbpl_tasks.py", line 38, in submit_bug_comment
    req.send_request()
  File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tbpl.py", line 163, in send_request
    r.raise_for_status()
  File "/usr/lib/python2.6/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 400 Client Error: Bad Request

-> This instance at least, will be fixed by bug 1078415

Comment 8

[Ed Morley [:emorley]]

Though we should probably:
a) add a comment length cap in the submit-bug-comment task itself anyway (eg in case any one comment line is excessively long, and so the total failure lines cap doesn't save us)
b) improve the annotation of failures in newrelic, eg:
https://docs.newrelic.com/docs/apm/other-features/attributes/collecting-custom-attributes
c) actually output the error text in the response from Bugzilla, in the celery log, rather than just the exception

Comment 9

[Ed Morley [:emorley]]

Once bug 1124842 is reviewed and in production, I'll be able to dig into this more.

Comment 10

[Ed Morley [:emorley]]

Bingo: starring the failure in comment 7 again, now more logging has landed...

[2015-01-30 08:27:07,628: ERROR/Worker-1] HTTPError 400 submitting to https://bugzilla.mozilla.org/rest/bug/1114541/comment: {"documentation":"http://www.bugzilla.org/docs/tip/en/html/api/","error":true,"code":114,"message":"Comments cannot be longer than 65535 characters."}

Comment 11

[Ed Morley [:emorley]]

Attachment: Bug comments: Limit the number of failures & total length

Comment 12

[Cameron Dawson [:camd]]

Comment on attachment 8563830 [details] [review]
Bug comments: Limit the number of failures & total length

looks good!

Comment 13

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/4e4ef16df59a459e9db63e4a229f633f76e96958
Bug 1114785 - Limit the total length of comments submitted to Bugzilla

Bugzilla comments can be no more than 65535 characters in length. To
both avoid hitting this limit (and the comment being rejected) and to
reduce the spam left on bugs, truncate the comment body at 40000
characters.