153637 - non patch attachments limit at 160K ????

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[Alexandru Savulov]

who had the ideea of limiting the non-patch attachment size to less than 160K
(or so)? 
what about complex packed testcases (containing html files, images and so on)?

are we supposed now to live with that for the rest of our life and add file
after file (instead of one zipped archive) having to deal with bugzilla
attachment href's for every reference in the testcase and so on?

das war keine gute idee leute!

Comment 1

[Bradley Baetz (:bbaetz)]

Its a meg by default. -> m.o/bmo issues

Comment 2

[Myk Melez [:myk] [@mykmelez]]

I did this last week when the attachments table was about to exceed 2GB in size,
which the filesystem would have rejected, causing errors and data loss when
anyone wanted to submit any attachment.  AOL IC has since remounted the
filesystem with large file support, so that's no longer a problem, but I still
don't want to increase the limit back to 1MB because it makes it way too easy
for users to attach huge uncompressed images and the like.

How large are your zip files likely to be?

Comment 3

[Alexandru Savulov]

well, is hard to say how large. the problem with the layout tescases is that
sometimes only a very special structure of the testcase will work and make us able 
to reproduce the bug. sometimes a small file will do, sometimes we must attach
entire pages with pictures and so on, sometimes a complex image (i.e. animated
gif with a lot of frames) will be needed, sometimes people try to attach
screenshots, to show us the problem. what were the biggest attachmets for layout
bugs so far? maybe you could set a selective attachment size. for layout bugs
bigger, for other bugs smaller.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

mysql> SELECT attachments.bug_id, attach_id, mimetype, length(thedata) FROM
bugs, attachments WHERE bugs.product = 'Browser' AND bugs.component = 'Layout'
AND bugs.bug_id = attachments.bug_id AND mimetype LIKE '%zip%' ORDER BY
length(thedata) DESC;
+--------+-----------+------------------------------+-----------------+
| bug_id | attach_id | mimetype                     | length(thedata) |
+--------+-----------+------------------------------+-----------------+
|  90294 |     42210 | application/x-zip-compressed |          561768 |
|  90294 |     42256 | application/x-zip-compressed |          473081 |
| 122638 |     67109 | application/zip              |          194815 |
|  88480 |     48358 | application/x-zip-compressed |          172592 |
| 121175 |     66333 | application/x-zip-compressed |          139985 |
|  71398 |     27293 | application/x-zip-compressed |          110697 |
|  66147 |     79379 | application/x-zip-compressed |           80714 |
|  13743 |      2895 | application/x-zip-compressed |           62349 |
| 152960 |     88469 | application/x-zip-compressed |           44081 |
|  96633 |     71193 | application/x-zip-compressed |           36936 |
| 134102 |     76680 | application/x-gzip           |           34802 |
| 112012 |     62176 | application/x-zip-compressed |           23552 |
| 112012 |     62774 | application/x-zip-compressed |           23552 |
| 115667 |     62410 | application/x-gzip           |           19557 |
|  84909 |     37883 | application/x-zip-compressed |           19515 |
|  21920 |      3911 | application/x-zip-compressed |           15787 |
| 110854 |     58447 | application/x-zip-compressed |           13216 |
|  46027 |     13251 | application/zip              |            8661 |
|  89951 |     43613 | application/x-gzip           |            5782 |
| 123895 |     68557 | application/zip              |            5770 |
|  42238 |     10075 | apllication/zip              |            4492 |
|  52265 |     25043 | application/x-zip-compressed |            3530 |
| 132058 |     74993 | application/x-zip-compressed |            2770 |
| 128345 |     86786 | application/x-zip-compressed |            2766 |
| 127883 |     71645 | application/x-zip-compressed |            2615 |
| 105652 |     54212 | application/x-zip-compressed |            2170 |
|  44263 |     10894 | application/x-zip            |            2096 |
| 136820 |     78694 | application/x-zip-compressed |            2047 |
| 134795 |     77195 | application/x-zip-compressed |            1265 |
| 110825 |     58395 | application/x-zip-compressed |             883 |
| 118915 |     65013 | application/x-gzip           |             836 |
| 127074 |     71190 | application/zip              |             822 |
|  52024 |     14336 | application/zip              |             668 |
|  43699 |     65742 | application/x-zip-compressed |             640 |
| 151649 |     87580 | application/zip              |             533 |
| 123962 |     73795 | application/zip              |             483 |
+--------+-----------+------------------------------+-----------------+
36 rows in set (22.48 sec)

mysql> SELECT attachments.bug_id, attach_id, mimetype, length(thedata) FROM
bugs, attachments WHERE bugs.product = 'Browser' AND bugs.component = 'Layout'
AND bugs.bug_id = attachments.bug_id ORDER BY length(thedata) DESC LIMIT 50;
+--------+-----------+------------------------------+-----------------+
| bug_id | attach_id | mimetype                     | length(thedata) |
+--------+-----------+------------------------------+-----------------+
|  99435 |     49193 | application/octet-stream     |         1470368 |
| 103473 |     53538 | text/plain                   |         1454346 |
| 103473 |     53913 | text/plain                   |         1454346 |
| 103473 |     53915 | text/plain                   |         1454346 |
| 103473 |     53918 | text/plain                   |         1454346 |
| 103473 |     52489 | application/octet-stream     |         1252320 |
| 103473 |     53540 | application/octet-stream     |         1250687 |
| 103473 |     52378 | application/octet-stream     |         1236145 |
| 103473 |     53539 | text/plain                   |         1165534 |
| 133512 |     76191 | image/bmp                    |          996406 |
| 133512 |     76192 | image/bmp                    |          996406 |
| 109885 |     57601 | text/html                    |          972205 |
| 145425 |     85405 | text/html                    |          888784 |
| 139986 |     81379 | text/html                    |          860149 |
|  25406 |      4645 | text/plain                   |          851660 |
|  98001 |     61440 | text/html                    |          838179 |
|  15116 |     75829 | application/octet-stream     |          808398 |
| 121330 |     66130 | text/html                    |          801714 |
|  33339 |     78448 | image/png                    |          791490 |
| 111667 |     62972 | text/html                    |          787473 |
| 123609 |     75679 | image/jpeg                   |          730505 |
| 130461 |     73834 | text/html                    |          714605 |
|  93758 |     68051 | text/html                    |          704543 |
| 133133 |     75893 | image/bmp                    |          699094 |
| 107733 |     62733 | text/html                    |          693406 |
| 107733 |     55900 | text/html                    |          677314 |
|  77282 |     31954 | image/jpeg                   |          671780 |
|  78645 |     39005 | text/plain                   |          669116 |
| 123609 |     75678 | image/jpeg                   |          652753 |
|  86947 |     49342 | text/html                    |          650561 |
|  47641 |     13516 | image/pict                   |          637048 |
| 107846 |     56057 | text/plain                   |          633784 |
|  36417 |      8855 | image/jpeg                   |          589202 |
| 115379 |     61836 | text/html                    |          576624 |
| 143097 |     83285 | text/html                    |          566766 |
|  90294 |     42210 | application/x-zip-compressed |          561768 |
|  60397 |     19423 | text/html                    |          543460 |
| 109885 |     64555 | text/html                    |          533704 |
| 107846 |     56011 | text/plain                   |          520779 |
| 127534 |     75441 | text/html                    |          499890 |
| 116437 |     65191 | text/html                    |          493148 |
|  13341 |      1605 | image/pict                   |          486134 |
|  13341 |      1826 | image/bmp                    |          481078 |
|  90294 |     42256 | application/x-zip-compressed |          473081 |
|  47641 |     12371 | image/pict                   |          468524 |
|  36417 |      8854 | image/jpeg                   |          449626 |
| 101223 |     50470 | image/png                    |          447258 |
|  99010 |     50834 | text/plain                   |          437674 |
| 107881 |     56051 | image/jpeg                   |          430263 |
|  98627 |     64451 | text/html                    |          429730 |
+--------+-----------+------------------------------+-----------------+
50 rows in set (13.33 sec)

Comment 5

[Alexandru Savulov]

well, AFAICT there are some big attachments that were needed in layout testing.
now is up to you guys to decide whether you want to rise that limit or not based
on the technical circumstances. i'm not aware of the hardware / systems limits
so i can just guess, but i think that 1M for layout bugs would be a reasonable
amount. from your querys the avg is somwhere around 0.3M - 0.5M considering that
there are a lot of smaller atachments too, however there are the peaks to that
range far over that avg.

Comment 6

[Roland Mainz]

The idea of the 150KB limit is not a good one - users cannot attach PostScript
jobs anymore since they usually exceed 150MB (even the compressed ones).
Result: We can't process/fix some bugs sinc we can't get the info from the
users. BAD.

Comment 7

[Myk Melez [:myk] [@mykmelez]]

150KB may be too low, but 1MB is certainly too high.  My random sampling of
large attachments on the list above shows 87-98% compression rates with zip. 
None of them compressed to larger than about 140KB.

Of course, compressing attachments before posting them is a pain in the butt; it
would be better if Bugzilla did it automatically.

Roland, which attachments are larger than 150KB compressed?

Comment 8

[Matthew Tuck [:CodeMachine]]

Myk, maybe this could be restricted based on MIME type.  If an attachment is a
recognised compressed file, the limit could be increased to 1Mb or whatever. 
Otherwise, 150K would be the limit.

Comment 9

[Roland Mainz]

The limit of 160KB is even to small for screenshots (GIF/PNG). Users have
started to move them to non-bugzilla places (where they usually disappear after
some time; and then developers/QA staff have to ask for new screenshots etc.
etc.).

Maybe we should change the summary of this bug to "attachment functionality of
bugzilla has become useless".

Comment 10

[Myk Melez [:myk] [@mykmelez]]

Can you give me an example of a screenshot for which 160K is too small?

Comment 11

[Roland Mainz]

Myk Melez wrote:
> Can you give me an example of a screenshot for which 160K is too small?

Do a simple calculation: Screenshot at 1024x768 - Mozilla viewing slashdot -
results in a GIF of 123250 bytes (~120KB).

(120KB / (1024*768)) * (1280*1024) = 200KB, therefore I guess at 1280*1024 you
should hit the 160KB barrier very easily (file size of compressed images does
not scale linear of the size of the uncompressed image).

I can provide samples next week on demand (as much you want) ...

Comment 12

[Gervase Markham [:gerv]]

The max size is now 500KB.

Gerv