[prod] ISE: IntegrityError: (1062, "Duplicate entry 'XcW' for key

VERIFIED FIXED in 1.1

Status

Websites
markup.mozilla.org
P1
normal
VERIFIED FIXED
7 years ago
7 years ago

People

(Reporter: mbrandt, Assigned: wenzel)

Tracking

unspecified

Details

(Whiteboard: [prod])

(Reporter)

Description

7 years ago
Traceback (most recent call last):

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/views/decorators/http.py", line 37, in inner
    return func(request, *args, **kwargs)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/markup/requests.py", line 125, in save_mark
    new_mark_reference = common.save_new_mark_with_data(mark_data, request.META['REMOTE_ADDR'])

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/../ffdemo/markup/common.py", line 73, in save_new_mark_with_data
    reference = create_save_mark(hash(stripped_points_obj_full), obscurred_ip, stripped_points_obj_simplified, data)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/../ffdemo/markup/common.py", line 104, in create_save_mark
    new_mark.save()

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/base.py", line 434, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/base.py", line 527, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/query.py", line 1479, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/sql/compiler.py", line 783, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/sql/compiler.py", line 727, in execute_sql
    cursor.execute(sql, params)

  File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/backends/mysql/base.py", line 86, in execute
    return self.cursor.execute(query, args)

  File "/usr/lib/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)

  File "/usr/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue

IntegrityError: (1062, "Duplicate entry 'XcW' for key 'markup_mark_reference_uniq'")
Assignee: nobody → jbresnik
Blocks: 628811

Comment 1

7 years ago
Yea this is related to the original we had - @wenzel , why is the transaction code removed in this commit:

https://github.com/mozilla/markup/commit/6ca2e833d837ef5ae62496044e630768677775c0

?

Comment 2

7 years ago
Also reference key are supposed to be 4 characters..   is there a way to recreate this?
(Reporter)

Comment 3

7 years ago
brez: we have access to the full traceback... would that help? I can email it to you.
(Reporter)

Comment 4

7 years ago
fyi, mpressman discovered that there are marks with as few as 2 characters. Are you sure this is not expected behavior? Can you check what the code is doing?
(Assignee)

Updated

7 years ago
Assignee: jbresnik → fwenzel
(Assignee)

Comment 5

7 years ago
The sequence generation uses an UPDATE followed by a SELECT which I suspect leads to the latter being directed at the slave, so it fails under load due to replication lag.

Using a cursor on the master only:

https://github.com/mozilla/markup/commit/fa30230
(Assignee)

Comment 6

7 years ago
(In reply to comment #5)
> The sequence generation uses an UPDATE followed by a SELECT which I suspect
> leads to the latter being directed at the slave, so it fails under load due
> to replication lag.

jbalogh points out that this is not the case, the cursor already points to the master only. So this patch does not harm, but it also doesn't fix the issue.
Priority: -- → P1
(Reporter)

Comment 7

7 years ago
Here's the current list of traceback email's denoting duplicate marks on prod:

    XcW
    MkPj
    uNTg
    rbAC
    uMs7
    YgfU
    dgHq
    Dmkq
    anVG
    r3kM
    RMuf
    PNyg
    2TwY
    mNAx
(Assignee)

Comment 8

7 years ago
mpressman, can you pastebin another list of the current duplicates we know of? We should try to extract a pattern here, and I currently don't know what these have in common.
http://it.pastebin.mozilla.org/1246278
(Assignee)

Comment 10

7 years ago
Thanks for helping figure this out, Matt! Apparently the references field is not case sensitive but should be. Added a migration for this here:

https://github.com/mozilla/markup/commit/57cb23a
Status: NEW → ASSIGNED
(Assignee)

Comment 11

7 years ago
Pushed to prod.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 12

7 years ago
Thx mpressman & wenzel for the detective work. What are the verification points on this other than perhaps just the lack of ISEs?
(Assignee)

Comment 13

7 years ago
Sadly, all we can do is verify by absence of ISEs. Considering we got about one per hour (right?) I think we can observe today and verify this by end of day if none reoccur.

Note that this is unrelated to the IOError ISEs that we also got on occasion.
(Reporter)

Comment 14

7 years ago
(In reply to comment #13)
> I think we can observe today and verify this by end of
> day if none reoccur.

I'm agreeable with that. We'll keep an eye on the ISEs and bump to verified at the end of the day.
(Reporter)

Comment 15

7 years ago
QA verifying (comment 13).  Tracebacks have ceased arriving in the inbox. Thx for the patch.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.