[heartbeat] MultipleObjectsReturned: get() returned more than one Answer -- it returned 2!

RESOLVED FIXED

Status

P1
normal
RESOLVED FIXED
4 years ago
2 years ago

People

(Reporter: willkg, Assigned: willkg)

Tracking

Details

(Whiteboard: u=gregg c=heartbeat p=2 s=input.2015q1)

Saw one traceback where it looks like we do a .get() for a flow_id that returns two records.

Needs investigation.

Traceback:

Internal Server Error: /api/v2/hb/
Traceback (most recent call last):
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/framework_django.py", line 497, in wrapper
    return wrapped(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/base/utils.py", line 457, in decorated_func
    response = f(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/component_djangorestframework.py", line 27, in _nr_wrapper_APIView_dispatch_
    return wrapped(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 403, in dispatch
    response = self.handle_exception(exc)
  File "/data/www/input.mozilla.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 400, in dispatch
    response = handler(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/heartbeat/views.py", line 58, in post
    flow_id=valid_data.flow_id
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 361, in get
    num if num <= MAX_GET_RESULTS else 'more than %s' % MAX_GET_RESULTS
MultipleObjectsReturned: get() returned more than one Answer -- it returned 2!

Also interesting is that the error email has this line in it:

POST:<could not parse>,

I suspect that's because it's not POST data but rather JSON data. We might need to tweak Django error handling so that it shows better errors for API-related requests.
Grabbing this to look into now.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
I spun the POST:<could not parse> bit into bug #1136840.

The bit of code is question is this:

            ans = Answer.objects.get(
                person_id=valid_data.person_id,
                survey_id=valid_data.survey_id,
                flow_id=valid_data.flow_id
            )

i.e. there are two hb answers with the same (person_id, survey_id, flow_id) combination.

There's no db integrity for making that tuple unique. It's entirely possible there's a race condition where two posts come in at the "same time" and both create answers in the db.

I think what we want to do is this:

1. write a data migration to delete any duplicates
2. add a unique_together thing to the model
3. change the post code so that if it hits an integrity error of this kind, it tries an update or something
In a PR: https://github.com/mozilla/fjord/pull/500

I skipped step 3 for now. I want to see what errors we get if we get any before I go fix them.
Whiteboard: u=gregg c=heartbeat p= s=input.2015q1 → u=gregg c=heartbeat p=2 s=input.2015q1
Pushed this to prod just now.

Pretty sure this should be fixed now. If there are more instances, I'll see them in the error email and figure out whether to create a new bug or reopen this one thing.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.