Closed Bug 1469777 Opened 6 years ago Closed 6 years ago

Failure summary not loading for some jobs due to HTTP 500 from /bug_suggestions/ - 'ProgrammingError: (1064, "syntax error, unexpected '-'")'

Categories

(Tree Management :: Treeherder, enhancement, P1)

Product:
Tree Management
Component:
Treeherder
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: apavel, Assigned: ghickman)

References

Details

Attachments

(5 files)

500 internal server error
119.79 KB, image/jpeg
Details
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3703
47 bytes, text/x-github-pull-request
Details | Review
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3722
47 bytes, text/x-github-pull-request
Details | Review
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3727
47 bytes, text/x-github-pull-request
Details | Review
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3751
47 bytes, text/x-github-pull-request
emorley
: review+
Details | Review
Example on New Relic: https://rpm.newrelic.com/accounts/677903/applications/14179757/filterable_errors?tw%5Bend%5D=1529491633&tw%5Bstart%5D=1529448433#/show/1404e87c-746e-11e8-b521-0242ac110006_0_6922/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart /treeherder.webapp.api.jobs:JobsViewSet.bug_suggestions URL /api/project/mozilla-inbound/jobs/183917498/bug_suggestions/ Traceback: ... File "/app/treeherder/webapp/api/jobs.py", line 443, in bug_suggestions File "/app/treeherder/model/error_summary.py", line 38, in get_error_summary File "/app/treeherder/model/error_summary.py", line 58, in bug_suggestions_line File "/app/treeherder/model/models.py", line 235, in search File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 1233, in __iter__ File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/query.py", line 80, in __iter__ File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/query.py", line 114, in _execute_query File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute File "/app/.heroku/python/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__ File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 101, in execute File "/app/.heroku/python/lib/python2.7/site-packages/newrelic/hooks/database_dbapi2.py", line 25, in execute File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/cursors.py", line 250, in execute File "/app/.heroku/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler django.db.utils:ProgrammingError: (1064, "syntax error, unexpected '-'") This seems related to bug 1224931. George I don't suppose you'd mind taking a look at this (pre-existing) issue?
Flags: needinfo?(ghickman)
Priority: -- → P2
See Also: → 1224931
Assignee: nobody → ghickman
Flags: needinfo?(ghickman)
Attachment #8987014 - Flags: review?(emorley)
Comment on attachment 8987014 [details] [review] Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3703 I've left a comment - thank you for tracking this down!
Attachment #8987014 - Flags: review?(emorley)
Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/aa36d30867f2e5f9958527bdde6554bcb6939038 Bug 1469777 - Catch and log errors when searching the Bugs Cache https://github.com/mozilla/treeherder/commit/fab08a077563e6f1f74c325c7d8775995f3caecb Bug 1469777 - Remove hyphens from search terms MySQL treats hyphens like a stop word (even though they aren't listed in `INFORMATION_SCHEMA.INNODB_FT_DEFAULT_STOPWORD`), causing a syntax error. We replace hyphens with a space so as not to break search for Sheriff's pasting errors into the bug summary.
This is fixed on master and deployed to stage, but the prod deploy is waiting on a few unrelated fixes (deps of bug 1450022).
Priority: P2 → P1
Summary: wpt jobs loading in a loop - 500 internal server error → Failure summary not loading for some jobs due to HTTP 500 from /bug_suggestions/ - 'ProgrammingError: (1064, "syntax error, unexpected '-'")'
New relic is reporting this on stage: AttributeError: 'module' object has no attribute 'record_exception' https://rpm.newrelic.com/accounts/677903/applications/14179733/filterable_errors?tw%5Bend%5D=1530024235&tw%5Bstart%5D=1530022435#/show/d0585598-794e-11e8-b521-0242ac110006_11144_18094/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart Looks like it's `newrelic.agent.record_exception()` rather than `newrelic.record_exception()` (sorry my example in the review feedback was just off the top of my head). Guess as well it means we don't have any test coverage of it. George, I don't suppose you'd mind taking a look?
Flags: needinfo?(ghickman)
Not at all, I should have paid closer attention!
Flags: needinfo?(ghickman)
After the comment 15 fix, on stage there is: /treeherder.webapp.api.jobs:JobsViewSet.bug_suggestions django.db.utils:ProgrammingError: (1064, "syntax error, unexpected '@', expecting $end") https://rpm.newrelic.com/accounts/677903/applications/14179733/filterable_errors?tw%5Bend%5D=1530105023&tw%5Bstart%5D=1530103223#/show/53b5e0fd-7a0b-11e8-b521-0242ac110006_4629_11527/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart
Flags: needinfo?(ghickman)
I assume we're getting the exception report because of the record_exception calls. However this prompted me to do some more digging and I think there's a way to sanitise the search strings with a regex. I'm going to run with that as it seems like a better way to handle this.
Flags: needinfo?(ghickman)
Yeah it's from record_exception - the New Relic properties lists response.status as 200, and trying the URL linked to that report shows the response minus bug suggestions, as expected. There are just more characters that need sanitation.
I'm now seeing a new exception on New Relic - "sre_constants:error: missing group name": https://rpm.newrelic.com/accounts/677903/applications/14179733/filterable_errors?tw%5Bend%5D=1530536203&tw%5Bstart%5D=1530534403#/show/208f5712-7df7-11e8-ba42-0242ac110009_0_4383/stack_trace?top_facet=transactionUiName&primary_facet=error.class&barchart=barchart It looks like the `re.sub()` arguments are inverted -- the 2nd argument should be the replacement character rather than the original string. Therefore there will never be any any search matches using this RE. It also means we have zero test coverage of this. Would you mind taking a look?
Flags: needinfo?(ghickman)
Sure thing, would be good to get some test coverage on that!
Flags: needinfo?(ghickman)
Attachment #8989372 - Flags: review?(emorley)
Comment on attachment 8989372 [details] [review] Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3751 Many thanks :-)
Attachment #8989372 - Flags: review?(emorley) → review+
There are no more exceptions on New Relic after this landed :-)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: Treeherder: Log Parsing & Classification → TreeHerder
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: