1136809 - [heartbeat] MultipleObjectsReturned: get() returned more than one Answer -- it returned 2!

Status: RESOLVED FIXED

(Input Graveyard :: Backend, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Saw one traceback where it looks like we do a .get() for a flow_id that returns two records.

Needs investigation.

Traceback:

Internal Server Error: /api/v2/hb/
Traceback (most recent call last):
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/framework_django.py", line 497, in wrapper
    return wrapped(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/base/utils.py", line 457, in decorated_func
    response = f(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/component_djangorestframework.py", line 27, in _nr_wrapper_APIView_dispatch_
    return wrapped(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 403, in dispatch
    response = self.handle_exception(exc)
  File "/data/www/input.mozilla.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 400, in dispatch
    response = handler(request, *args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/heartbeat/views.py", line 58, in post
    flow_id=valid_data.flow_id
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 361, in get
    num if num <= MAX_GET_RESULTS else 'more than %s' % MAX_GET_RESULTS
MultipleObjectsReturned: get() returned more than one Answer -- it returned 2!

Also interesting is that the error email has this line in it:

POST:<could not parse>,

I suspect that's because it's not POST data but rather JSON data. We might need to tweak Django error handling so that it shows better errors for API-related requests.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to look into now.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I spun the POST:<could not parse> bit into bug #1136840.

The bit of code is question is this:

            ans = Answer.objects.get(
                person_id=valid_data.person_id,
                survey_id=valid_data.survey_id,
                flow_id=valid_data.flow_id
            )

i.e. there are two hb answers with the same (person_id, survey_id, flow_id) combination.

There's no db integrity for making that tuple unique. It's entirely possible there's a race condition where two posts come in at the "same time" and both create answers in the db.

I think what we want to do is this:

1. write a data migration to delete any duplicates
2. add a unique_together thing to the model
3. change the post code so that if it hits an integrity error of this kind, it tries an update or something

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

In a PR: https://github.com/mozilla/fjord/pull/500

I skipped step 3 for now. I want to see what errors we get if we get any before I go fix them.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master: https://github.com/mozilla/fjord/commit/2d3d05ae430946adda30fe09782df970fbc01bde

Will push out in the next deploy.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pushed this to prod just now.

Pretty sure this should be fixed now. If there are more instances, I'll see them in the error email and figure out whether to create a new bug or reopen this one thing.