662830 - [prod] ISE: IntegrityError: (1062, "Duplicate entry 'XcW' for key

Status: VERIFIED FIXED

(Websites Graveyard :: markup.mozilla.org, defect, P1)

Description

[Matt Brandt [:mbrandt]]

Traceback (most recent call last):

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/views/decorators/http.py", line 37, in inner
    return func(request, *args, **kwargs)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/markup/requests.py", line 125, in save_mark
    new_mark_reference = common.save_new_mark_with_data(mark_data, request.META['REMOTE_ADDR'])

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/../ffdemo/markup/common.py", line 73, in save_new_mark_with_data
    reference = create_save_mark(hash(stripped_points_obj_full), obscurred_ip, stripped_points_obj_simplified, data)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/../ffdemo/markup/common.py", line 104, in create_save_mark
    new_mark.save()

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/base.py", line 434, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/base.py", line 527, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/query.py", line 1479, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/sql/compiler.py", line 783, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/sql/compiler.py", line 727, in execute_sql
    cursor.execute(sql, params)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/backends/mysql/base.py", line 86, in execute
    return self.cursor.execute(query, args)

  File "/usr/lib/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)

  File "/usr/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue

IntegrityError: (1062, "Duplicate entry 'XcW' for key 'markup_mark_reference_uniq'")

Comment 1

[[:brez]]

Yea this is related to the original we had - @wenzel , why is the transaction code removed in this commit:

https://github.com/mozilla/markup/commit/6ca2e833d837ef5ae62496044e630768677775c0

?

Comment 2

[[:brez]]

Also reference key are supposed to be 4 characters..   is there a way to recreate this?

Comment 3

[Matt Brandt [:mbrandt]]

brez: we have access to the full traceback... would that help? I can email it to you.

Comment 4

[Matt Brandt [:mbrandt]]

fyi, mpressman discovered that there are marks with as few as 2 characters. Are you sure this is not expected behavior? Can you check what the code is doing?

Comment 5

[Fred Wenzel [:wenzel]]

The sequence generation uses an UPDATE followed by a SELECT which I suspect leads to the latter being directed at the slave, so it fails under load due to replication lag.

Using a cursor on the master only:

https://github.com/mozilla/markup/commit/fa30230

Comment 6

[Fred Wenzel [:wenzel]]

(In reply to comment #5)
> The sequence generation uses an UPDATE followed by a SELECT which I suspect
> leads to the latter being directed at the slave, so it fails under load due
> to replication lag.

jbalogh points out that this is not the case, the cursor already points to the master only. So this patch does not harm, but it also doesn't fix the issue.

Comment 7

[Matt Brandt [:mbrandt]]

Here's the current list of traceback email's denoting duplicate marks on prod:

    XcW
    MkPj
    uNTg
    rbAC
    uMs7
    YgfU
    dgHq
    Dmkq
    anVG
    r3kM
    RMuf
    PNyg
    2TwY
    mNAx

Comment 8

[Fred Wenzel [:wenzel]]

mpressman, can you pastebin another list of the current duplicates we know of? We should try to extract a pattern here, and I currently don't know what these have in common.

Comment 9

[Matt Pressman [:mpressman]]

http://it.pastebin.mozilla.org/1246278

Comment 10

[Fred Wenzel [:wenzel]]

Thanks for helping figure this out, Matt! Apparently the references field is not case sensitive but should be. Added a migration for this here:

https://github.com/mozilla/markup/commit/57cb23a

Comment 11

[Fred Wenzel [:wenzel]]

Pushed to prod.

Comment 12

[Matt Brandt [:mbrandt]]

Thx mpressman & wenzel for the detective work. What are the verification points on this other than perhaps just the lack of ISEs?

Comment 13

[Fred Wenzel [:wenzel]]

Sadly, all we can do is verify by absence of ISEs. Considering we got about one per hour (right?) I think we can observe today and verify this by end of day if none reoccur.

Note that this is unrelated to the IOError ISEs that we also got on occasion.

Comment 14

[Matt Brandt [:mbrandt]]

(In reply to comment #13)
> I think we can observe today and verify this by end of
> day if none reoccur.

I'm agreeable with that. We'll keep an eye on the ISEs and bump to verified at the end of the day.

Comment 15

[Matt Brandt [:mbrandt]]

QA verifying (comment 13).  Tracebacks have ceased arriving in the inbox. Thx for the patch.