Closed Bug 981714 Opened 6 years ago Closed 6 years ago

unable to attach a file to bugzilla (duplicate primary key)

Categories

(Data & BI Services Team :: DB: MySQL, task, critical)

x86
macOS
task
Not set
critical

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: glob, Assigned: bjohnson)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

attaching any file to bugzilla is currently failing:

DBD::mysql::db do failed: Duplicate entry '8388607' for key 'PRIMARY' [for Statement "INSERT INTO attachments (ispatch, bug_id, description, filename, isprivate, submitter_id, creation_ts, mimetype, modification_time) VALUES (?,?,?,?,?,?,?,?,?)"] at /data/www/bugzilla.mozilla.org/Bugzilla/Object.pm line 696.

it's always the same attach_id.
Summary: unable to attach a file to bugzilla (duplicate primay key) → unable to attach a file to bugzilla (duplicate primary key)
The attachment ID column was set to be a signed mediumint. Unfortunately, we expanded to the maximum value of a signed mediumint (8388607) as per: https://dev.mysql.com/doc/refman/5.5/en/integer-types.html

The only solution here is to expand that column. I've started the alter table statement to expand it into an unsigned bigint (the largest integer data type currently available).

Unfortunately, we're at the mercy of the system until it completes and no new attachments will be able to be added.
Flags: needinfo?(craigcook.bugz)
Craig Cook is on PTO all this week. Why was he needinfo'd here?
Flags: needinfo?(craigcook.bugz)
Attached file hands in the air! \o/
Please work!
This was resolved about 20 minutes ago.
Things are a bit slow and will be for about 20 minutes longer.
Now that the immediate problem is solved, has anyone made sure there aren't any code dependencies on field type? For example, I would hate to have some bounds checking code break mysteriously because it still expected the field to be of type Medium Int. If this doesn't apply then just call me paranoid. Bob
Something is still wrong.  I repeatedly get "Error 500" from Bugzilla when trying to
request approval-aurora on the patches in bug 978001.
Yep, I still can't upload an attachment.. Even trying to comment in bug 922680 resulted in:
DBD::mysql::db do failed: Lock wait timeout exceeded; try restarting transaction [for Statement "INSERT INTO flags (status, bug_id, setter_id, attach_id, requestee_id, modification_date, creation_date, type_id) VALUES (?,?,?,?,?,?,?,?)"]
Lock wait timeouts will continue until morale improves...


But in all seriousness, The issue happened across the board. We corrected it on the master around 12:10 PDT today. However, because all of the slaves had to also correct the issue, we were significantly reduced in power until the slaves caught up (1/4 bugzilla machines doing their job successfully). It has been a significantly slow experience and we apologize for the delay.

However, 2 more slaves have just been introduced and we're back to 3/4 machines doing their job. Speed should start to improve until the 4th is added, which should be done in the next hour.

Thank you.
For anyone having issues with attachments or bugs, please update/attach now and if you're still having issues, report it here. :)
Duplicate of this bug: 981833
I'm still having issues. I ended up creating 4 duplicates of a bug - bug 981799, bug 981813, bug 981819, and bug 981821. (It created it, but never managed to return the results page.) That part can be explained by the slowdown/overload. (A bunch of other people did the same thing, as you can see if you sort the list of bugs posted in the last hour or two by summary.) 

But much more recently, I went in and marked the latter 3 as RESOLVED/INVALID. That status now shows up in the (inline) history and the bug header, but a search still shows them as being NEW. So something's inconsistent. Maybe it's just caching for the searches?

Bug 981813 is an example of such a bug.
Steve, that's actually consistent not with this issue but with bug 974338. There are periodic replication spikes that we're solving in bugzilla.
(In reply to Bob Moss :bmoss from comment #6)
> Now that the immediate problem is solved, has anyone made sure there aren't
> any code dependencies on field type?

yes, i checked the code last night and we're ok there.

bugzilla does have an internal representation of its schema in its database abstraction layer which will need to be updated to reflect this change; that's bug 981756.
(In reply to Byron Jones ‹:glob› from comment #14)
> (In reply to Bob Moss :bmoss from comment #6)
> > Now that the immediate problem is solved, has anyone made sure there aren't
> > any code dependencies on field type?
> 
> yes, i checked the code last night and we're ok there.
> 
> bugzilla does have an internal representation of its schema in its database
> abstraction layer which will need to be updated to reflect this change;
> that's bug 981756.

That's great. Thanks for checking.
For reference, we changed these fields on all production machines, including backups:
+-------------------------+---------------+
| table_name              | column_name   |
+-------------------------+---------------+
| attachments             | attach_id     |
| autoland_attachments    | attach_id     |
| bugs_activity           | attach_id     |
| flag_state_activity     | attachment_id |
| flags                   | attach_id     |
| sanitized_bugs_activity | attach_id     |
+-------------------------+---------------+
What's left to do, in this bug?

Changing stage is bug 981756. 

We fixed the attachment IDs that were set to the max(mediumint) in bug 981986.

Monitoring integer types to make sure we're aware of this before we hit the problem again is in bug 981841.

If that's everything, let's resolve this bug.
Blocks: 981756, 981986, 981841
I believe we're resolved.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.