Traceback (most recent call last): File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/core/handlers/base.py", line 100, in get_response response = callback(request, *callback_args, **callback_kwargs) File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/views/decorators/http.py", line 37, in inner return func(request, *args, **kwargs) File "/data/www/python/markup.mozilla.org/markup/ffdemo/markup/requests.py", line 125, in save_mark new_mark_reference = common.save_new_mark_with_data(mark_data, request.META['REMOTE_ADDR']) File "/data/www/python/markup.mozilla.org/markup/ffdemo/../ffdemo/markup/common.py", line 73, in save_new_mark_with_data reference = create_save_mark(hash(stripped_points_obj_full), obscurred_ip, stripped_points_obj_simplified, data) File "/data/www/python/markup.mozilla.org/markup/ffdemo/../ffdemo/markup/common.py", line 104, in create_save_mark new_mark.save() File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/base.py", line 434, in save self.save_base(using=using, force_insert=force_insert, force_update=force_update) File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/base.py", line 527, in save_base result = manager._insert(values, return_id=update_pk, using=using) File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/manager.py", line 195, in _insert return insert_query(self.model, values, **kwargs) File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/query.py", line 1479, in insert_query return query.get_compiler(using=using).execute_sql(return_id) File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/sql/compiler.py", line 783, in execute_sql cursor = super(SQLInsertCompiler, self).execute_sql(None) File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/models/sql/compiler.py", line 727, in execute_sql cursor.execute(sql, params) File "/data/www/python/markup.mozilla.org/markup/ffdemo/vendor/src/django/django/db/backends/mysql/base.py", line 86, in execute return self.cursor.execute(query, args) File "/usr/lib/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute self.errorhandler(self, exc, value) File "/usr/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue IntegrityError: (1062, "Duplicate entry 'XcW' for key 'markup_mark_reference_uniq'")
Yea this is related to the original we had - @wenzel , why is the transaction code removed in this commit: https://github.com/mozilla/markup/commit/6ca2e833d837ef5ae62496044e630768677775c0 ?
Also reference key are supposed to be 4 characters.. is there a way to recreate this?
brez: we have access to the full traceback... would that help? I can email it to you.
fyi, mpressman discovered that there are marks with as few as 2 characters. Are you sure this is not expected behavior? Can you check what the code is doing?
The sequence generation uses an UPDATE followed by a SELECT which I suspect leads to the latter being directed at the slave, so it fails under load due to replication lag. Using a cursor on the master only: https://github.com/mozilla/markup/commit/fa30230
(In reply to comment #5) > The sequence generation uses an UPDATE followed by a SELECT which I suspect > leads to the latter being directed at the slave, so it fails under load due > to replication lag. jbalogh points out that this is not the case, the cursor already points to the master only. So this patch does not harm, but it also doesn't fix the issue.
Here's the current list of traceback email's denoting duplicate marks on prod: XcW MkPj uNTg rbAC uMs7 YgfU dgHq Dmkq anVG r3kM RMuf PNyg 2TwY mNAx
mpressman, can you pastebin another list of the current duplicates we know of? We should try to extract a pattern here, and I currently don't know what these have in common.
Thanks for helping figure this out, Matt! Apparently the references field is not case sensitive but should be. Added a migration for this here: https://github.com/mozilla/markup/commit/57cb23a
Pushed to prod.
Thx mpressman & wenzel for the detective work. What are the verification points on this other than perhaps just the lack of ISEs?
Sadly, all we can do is verify by absence of ISEs. Considering we got about one per hour (right?) I think we can observe today and verify this by end of day if none reoccur. Note that this is unrelated to the IOError ISEs that we also got on occasion.
(In reply to comment #13) > I think we can observe today and verify this by end of > day if none reoccur. I'm agreeable with that. We'll keep an eye on the ISEs and bump to verified at the end of the day.
QA verifying (comment 13). Tracebacks have ceased arriving in the inbox. Thx for the patch.