1180570 - store attachment size in the database

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[:glob ✱]

currently the size of an attachment is generated on demand from either the length of the blob or the size of the file on disk.

in order to support s3 we need the attachment size to be a column in attachments table.

Comment 1

[:glob ✱]

Attachment: 1180570_1.patch

- adds attach_size column to attachments table
- updates attachment object to set/get attach_size column
- removes unnecessary preloading of attachment sizes

Comment 2

[:glob ✱]

Attachment: 1180570_2.patch

- adds attach_size column to attachments table
- updates attachment object to set/get attach_size column
- removes unnecessary preloading of attachment sizes
- updates sanitizeme.pl to update attach_size when nuking attach data

Comment 3

[David Lawrence [:dkl]]

Comment on attachment 8629800 [details] [diff] [review]
1180570_2.patch

Review of attachment 8629800 [details] [diff] [review]:
-----------------------------------------------------------------

Minor (fix on commit)

1. contrib/sanitizeme.pl moved to scripts/

patching 1180570 with #8629800
fetching attachment #8629800 [details] [diff] [review]
Bug 1180570: 1180570_2.patch
creating 1180570-8629800.patch
Checking patch Bugzilla/Attachment.pm...
Checking patch Bugzilla/DB/Schema.pm...
Checking patch Bugzilla/Install/DB.pm...
Checking patch contrib/sanitizeme.pl...
error: contrib/sanitizeme.pl: No such file or directory

2. extensions/Push/t/ReviewBoard.t (not sure we even use this)

line 102:
                                    
filesize => length $data,

Otherwise works and looks good. r=dkl

Comment 4

[:glob ✱]

schema-only:

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   cd60ea7..28ac958  master -> master

note to self: re-run attachment size setter after pushing 2nd part to fix attachments uploaded between pushes.

UPDATE attachments
       INNER JOIN attach_data ON attach_data.id = attachments.attach_id
   SET attachments.attach_size = LENGTH(attach_data.thedata)

Comment 5

[:glob ✱]

i'm concerned that the population of the attachments.attach_size column make take a long time, resulting in a very long checksetup run.

i've disabled that part of the schema upgrade:

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   da4255a..940d6d4  master -> master

and i'll manually run it after the changes have been pushed.  this gives me much more flexibility with regards to gauging its impact and running it in batches.

Comment 6

[:glob ✱]

attachments.attach_size took ~10 minutes to populate, no errors reported while it was running.

> select sum(attach_size) from attachments
> 119291581473 (~120gb in 1,131,003 attachments)

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   425d780..2bd3296  master -> master