Closed Bug 1136809 Opened 10 years ago Closed 10 years ago

[heartbeat] MultipleObjectsReturned: get() returned more than one Answer -- it returned 2!

Categories

(Input Graveyard :: Backend, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

Details

(Whiteboard: u=gregg c=heartbeat p=2 s=input.2015q1)

Saw one traceback where it looks like we do a .get() for a flow_id that returns two records. Needs investigation. Traceback: Internal Server Error: /api/v2/hb/ Traceback (most recent call last): File "/data/www/input.mozilla.org/input/vendor/src/django/django/core/handlers/base.py", line 111, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/framework_django.py", line 497, in wrapper return wrapped(*args, **kwargs) File "/data/www/input.mozilla.org/input/fjord/base/utils.py", line 457, in decorated_func response = f(request, *args, **kwargs) File "/data/www/input.mozilla.org/input/vendor/src/django/django/views/decorators/csrf.py", line 57, in wrapped_view return view_func(*args, **kwargs) File "/data/www/input.mozilla.org/input/vendor/src/django/django/views/generic/base.py", line 69, in view return self.dispatch(request, *args, **kwargs) File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/component_djangorestframework.py", line 27, in _nr_wrapper_APIView_dispatch_ return wrapped(*args, **kwargs) File "/data/www/input.mozilla.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 403, in dispatch response = self.handle_exception(exc) File "/data/www/input.mozilla.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 400, in dispatch response = handler(request, *args, **kwargs) File "/data/www/input.mozilla.org/input/fjord/heartbeat/views.py", line 58, in post flow_id=valid_data.flow_id File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/manager.py", line 92, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 361, in get num if num <= MAX_GET_RESULTS else 'more than %s' % MAX_GET_RESULTS MultipleObjectsReturned: get() returned more than one Answer -- it returned 2! Also interesting is that the error email has this line in it: POST:<could not parse>, I suspect that's because it's not POST data but rather JSON data. We might need to tweak Django error handling so that it shows better errors for API-related requests.
Grabbing this to look into now.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
I spun the POST:<could not parse> bit into bug #1136840. The bit of code is question is this: ans = Answer.objects.get( person_id=valid_data.person_id, survey_id=valid_data.survey_id, flow_id=valid_data.flow_id ) i.e. there are two hb answers with the same (person_id, survey_id, flow_id) combination. There's no db integrity for making that tuple unique. It's entirely possible there's a race condition where two posts come in at the "same time" and both create answers in the db. I think what we want to do is this: 1. write a data migration to delete any duplicates 2. add a unique_together thing to the model 3. change the post code so that if it hits an integrity error of this kind, it tries an update or something
In a PR: https://github.com/mozilla/fjord/pull/500 I skipped step 3 for now. I want to see what errors we get if we get any before I go fix them.
Whiteboard: u=gregg c=heartbeat p= s=input.2015q1 → u=gregg c=heartbeat p=2 s=input.2015q1
Pushed this to prod just now. Pretty sure this should be fixed now. If there are more instances, I'll see them in the error email and figure out whether to create a new bug or reopen this one thing.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.