store attachment size in the database

RESOLVED FIXED

Status

()

bugzilla.mozilla.org
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: glob, Assigned: glob)

Tracking

Production
Dependency tree / graph

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Assignee)

Description

3 years ago
currently the size of an attachment is generated on demand from either the length of the blob or the size of the file on disk.

in order to support s3 we need the attachment size to be a column in attachments table.
(Assignee)

Comment 1

3 years ago
Created attachment 8629798 [details] [diff] [review]
1180570_1.patch

- adds attach_size column to attachments table
- updates attachment object to set/get attach_size column
- removes unnecessary preloading of attachment sizes
Attachment #8629798 - Flags: review?(dkl)
(Assignee)

Comment 2

3 years ago
Created attachment 8629800 [details] [diff] [review]
1180570_2.patch

- adds attach_size column to attachments table
- updates attachment object to set/get attach_size column
- removes unnecessary preloading of attachment sizes
- updates sanitizeme.pl to update attach_size when nuking attach data
Attachment #8629798 - Attachment is obsolete: true
Attachment #8629798 - Flags: review?(dkl)
Attachment #8629800 - Flags: review?(dkl)
(Assignee)

Updated

3 years ago
Blocks: 1180572
Comment on attachment 8629800 [details] [diff] [review]
1180570_2.patch

Review of attachment 8629800 [details] [diff] [review]:
-----------------------------------------------------------------

Minor (fix on commit)

1. contrib/sanitizeme.pl moved to scripts/

patching 1180570 with #8629800
fetching attachment #8629800 [details] [diff] [review]
Bug 1180570: 1180570_2.patch
creating 1180570-8629800.patch
Checking patch Bugzilla/Attachment.pm...
Checking patch Bugzilla/DB/Schema.pm...
Checking patch Bugzilla/Install/DB.pm...
Checking patch contrib/sanitizeme.pl...
error: contrib/sanitizeme.pl: No such file or directory

2. extensions/Push/t/ReviewBoard.t (not sure we even use this)

line 102:
                                    
filesize => length $data,

Otherwise works and looks good. r=dkl
Attachment #8629800 - Flags: review?(dkl) → review+
(Assignee)

Comment 4

3 years ago
schema-only:

To ssh://gitolite3@git.mozilla.org/webtools/bmo/bugzilla.git
   cd60ea7..28ac958  master -> master

note to self: re-run attachment size setter after pushing 2nd part to fix attachments uploaded between pushes.

UPDATE attachments
       INNER JOIN attach_data ON attach_data.id = attachments.attach_id
   SET attachments.attach_size = LENGTH(attach_data.thedata)
(Assignee)

Comment 5

3 years ago
i'm concerned that the population of the attachments.attach_size column make take a long time, resulting in a very long checksetup run.

i've disabled that part of the schema upgrade:

To ssh://gitolite3@git.mozilla.org/webtools/bmo/bugzilla.git
   da4255a..940d6d4  master -> master

and i'll manually run it after the changes have been pushed.  this gives me much more flexibility with regards to gauging its impact and running it in batches.
(Assignee)

Comment 6

3 years ago
attachments.attach_size took ~10 minutes to populate, no errors reported while it was running.

> select sum(attach_size) from attachments
> 119291581473 (~120gb in 1,131,003 attachments)

To ssh://gitolite3@git.mozilla.org/webtools/bmo/bugzilla.git
   425d780..2bd3296  master -> master
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.