non patch attachments limit at 160K ????

RESOLVED FIXED

Status

()

bugzilla.mozilla.org
General
RESOLVED FIXED
16 years ago
7 years ago

People

(Reporter: Alexandru Savulov, Assigned: Dawn Endico)

Tracking

Details

(Reporter)

Description

16 years ago
who had the ideea of limiting the non-patch attachment size to less than 160K
(or so)? 
what about complex packed testcases (containing html files, images and so on)?

are we supposed now to live with that for the rest of our life and add file
after file (instead of one zipped archive) having to deal with bugzilla
attachment href's for every reference in the testcase and so on?

das war keine gute idee leute!
Its a meg by default. -> m.o/bmo issues
Assignee: myk → endico
Component: Creating/Changing Bugs → Bugzilla: Other moz.org Issues
Product: Bugzilla → mozilla.org
QA Contact: matty → myk
Summary: non patch attachments limit at 160K ???? → non patch attachments limit at 160K ????
Version: unspecified → other
I did this last week when the attachments table was about to exceed 2GB in size,
which the filesystem would have rejected, causing errors and data loss when
anyone wanted to submit any attachment.  AOL IC has since remounted the
filesystem with large file support, so that's no longer a problem, but I still
don't want to increase the limit back to 1MB because it makes it way too easy
for users to attach huge uncompressed images and the like.

How large are your zip files likely to be?
(Reporter)

Comment 3

16 years ago
well, is hard to say how large. the problem with the layout tescases is that
sometimes only a very special structure of the testcase will work and make us able 
to reproduce the bug. sometimes a small file will do, sometimes we must attach
entire pages with pictures and so on, sometimes a complex image (i.e. animated
gif with a lot of frames) will be needed, sometimes people try to attach
screenshots, to show us the problem. what were the biggest attachmets for layout
bugs so far? maybe you could set a selective attachment size. for layout bugs
bigger, for other bugs smaller.
mysql> SELECT attachments.bug_id, attach_id, mimetype, length(thedata) FROM
bugs, attachments WHERE bugs.product = 'Browser' AND bugs.component = 'Layout'
AND bugs.bug_id = attachments.bug_id AND mimetype LIKE '%zip%' ORDER BY
length(thedata) DESC;
+--------+-----------+------------------------------+-----------------+
| bug_id | attach_id | mimetype                     | length(thedata) |
+--------+-----------+------------------------------+-----------------+
|  90294 |     42210 | application/x-zip-compressed |          561768 |
|  90294 |     42256 | application/x-zip-compressed |          473081 |
| 122638 |     67109 | application/zip              |          194815 |
|  88480 |     48358 | application/x-zip-compressed |          172592 |
| 121175 |     66333 | application/x-zip-compressed |          139985 |
|  71398 |     27293 | application/x-zip-compressed |          110697 |
|  66147 |     79379 | application/x-zip-compressed |           80714 |
|  13743 |      2895 | application/x-zip-compressed |           62349 |
| 152960 |     88469 | application/x-zip-compressed |           44081 |
|  96633 |     71193 | application/x-zip-compressed |           36936 |
| 134102 |     76680 | application/x-gzip           |           34802 |
| 112012 |     62176 | application/x-zip-compressed |           23552 |
| 112012 |     62774 | application/x-zip-compressed |           23552 |
| 115667 |     62410 | application/x-gzip           |           19557 |
|  84909 |     37883 | application/x-zip-compressed |           19515 |
|  21920 |      3911 | application/x-zip-compressed |           15787 |
| 110854 |     58447 | application/x-zip-compressed |           13216 |
|  46027 |     13251 | application/zip              |            8661 |
|  89951 |     43613 | application/x-gzip           |            5782 |
| 123895 |     68557 | application/zip              |            5770 |
|  42238 |     10075 | apllication/zip              |            4492 |
|  52265 |     25043 | application/x-zip-compressed |            3530 |
| 132058 |     74993 | application/x-zip-compressed |            2770 |
| 128345 |     86786 | application/x-zip-compressed |            2766 |
| 127883 |     71645 | application/x-zip-compressed |            2615 |
| 105652 |     54212 | application/x-zip-compressed |            2170 |
|  44263 |     10894 | application/x-zip            |            2096 |
| 136820 |     78694 | application/x-zip-compressed |            2047 |
| 134795 |     77195 | application/x-zip-compressed |            1265 |
| 110825 |     58395 | application/x-zip-compressed |             883 |
| 118915 |     65013 | application/x-gzip           |             836 |
| 127074 |     71190 | application/zip              |             822 |
|  52024 |     14336 | application/zip              |             668 |
|  43699 |     65742 | application/x-zip-compressed |             640 |
| 151649 |     87580 | application/zip              |             533 |
| 123962 |     73795 | application/zip              |             483 |
+--------+-----------+------------------------------+-----------------+
36 rows in set (22.48 sec)

mysql> SELECT attachments.bug_id, attach_id, mimetype, length(thedata) FROM
bugs, attachments WHERE bugs.product = 'Browser' AND bugs.component = 'Layout'
AND bugs.bug_id = attachments.bug_id ORDER BY length(thedata) DESC LIMIT 50;
+--------+-----------+------------------------------+-----------------+
| bug_id | attach_id | mimetype                     | length(thedata) |
+--------+-----------+------------------------------+-----------------+
|  99435 |     49193 | application/octet-stream     |         1470368 |
| 103473 |     53538 | text/plain                   |         1454346 |
| 103473 |     53913 | text/plain                   |         1454346 |
| 103473 |     53915 | text/plain                   |         1454346 |
| 103473 |     53918 | text/plain                   |         1454346 |
| 103473 |     52489 | application/octet-stream     |         1252320 |
| 103473 |     53540 | application/octet-stream     |         1250687 |
| 103473 |     52378 | application/octet-stream     |         1236145 |
| 103473 |     53539 | text/plain                   |         1165534 |
| 133512 |     76191 | image/bmp                    |          996406 |
| 133512 |     76192 | image/bmp                    |          996406 |
| 109885 |     57601 | text/html                    |          972205 |
| 145425 |     85405 | text/html                    |          888784 |
| 139986 |     81379 | text/html                    |          860149 |
|  25406 |      4645 | text/plain                   |          851660 |
|  98001 |     61440 | text/html                    |          838179 |
|  15116 |     75829 | application/octet-stream     |          808398 |
| 121330 |     66130 | text/html                    |          801714 |
|  33339 |     78448 | image/png                    |          791490 |
| 111667 |     62972 | text/html                    |          787473 |
| 123609 |     75679 | image/jpeg                   |          730505 |
| 130461 |     73834 | text/html                    |          714605 |
|  93758 |     68051 | text/html                    |          704543 |
| 133133 |     75893 | image/bmp                    |          699094 |
| 107733 |     62733 | text/html                    |          693406 |
| 107733 |     55900 | text/html                    |          677314 |
|  77282 |     31954 | image/jpeg                   |          671780 |
|  78645 |     39005 | text/plain                   |          669116 |
| 123609 |     75678 | image/jpeg                   |          652753 |
|  86947 |     49342 | text/html                    |          650561 |
|  47641 |     13516 | image/pict                   |          637048 |
| 107846 |     56057 | text/plain                   |          633784 |
|  36417 |      8855 | image/jpeg                   |          589202 |
| 115379 |     61836 | text/html                    |          576624 |
| 143097 |     83285 | text/html                    |          566766 |
|  90294 |     42210 | application/x-zip-compressed |          561768 |
|  60397 |     19423 | text/html                    |          543460 |
| 109885 |     64555 | text/html                    |          533704 |
| 107846 |     56011 | text/plain                   |          520779 |
| 127534 |     75441 | text/html                    |          499890 |
| 116437 |     65191 | text/html                    |          493148 |
|  13341 |      1605 | image/pict                   |          486134 |
|  13341 |      1826 | image/bmp                    |          481078 |
|  90294 |     42256 | application/x-zip-compressed |          473081 |
|  47641 |     12371 | image/pict                   |          468524 |
|  36417 |      8854 | image/jpeg                   |          449626 |
| 101223 |     50470 | image/png                    |          447258 |
|  99010 |     50834 | text/plain                   |          437674 |
| 107881 |     56051 | image/jpeg                   |          430263 |
|  98627 |     64451 | text/html                    |          429730 |
+--------+-----------+------------------------------+-----------------+
50 rows in set (13.33 sec)

(Reporter)

Comment 5

16 years ago
well, AFAICT there are some big attachments that were needed in layout testing.
now is up to you guys to decide whether you want to rise that limit or not based
on the technical circumstances. i'm not aware of the hardware / systems limits
so i can just guess, but i think that 1M for layout bugs would be a reasonable
amount. from your querys the avg is somwhere around 0.3M - 0.5M considering that
there are a lot of smaller atachments too, however there are the peaks to that
range far over that avg.

Comment 6

16 years ago
The idea of the 150KB limit is not a good one - users cannot attach PostScript
jobs anymore since they usually exceed 150MB (even the compressed ones).
Result: We can't process/fix some bugs sinc we can't get the info from the
users. BAD.
150KB may be too low, but 1MB is certainly too high.  My random sampling of
large attachments on the list above shows 87-98% compression rates with zip. 
None of them compressed to larger than about 140KB.

Of course, compressing attachments before posting them is a pain in the butt; it
would be better if Bugzilla did it automatically.

Roland, which attachments are larger than 150KB compressed?
Myk, maybe this could be restricted based on MIME type.  If an attachment is a
recognised compressed file, the limit could be increased to 1Mb or whatever. 
Otherwise, 150K would be the limit.

Comment 9

16 years ago
The limit of 160KB is even to small for screenshots (GIF/PNG). Users have
started to move them to non-bugzilla places (where they usually disappear after
some time; and then developers/QA staff have to ask for new screenshots etc.
etc.).

Maybe we should change the summary of this bug to "attachment functionality of
bugzilla has become useless".
Can you give me an example of a screenshot for which 160K is too small?

Comment 11

16 years ago
Myk Melez wrote:
> Can you give me an example of a screenshot for which 160K is too small?

Do a simple calculation: Screenshot at 1024x768 - Mozilla viewing slashdot -
results in a GIF of 123250 bytes (~120KB).

(120KB / (1024*768)) * (1280*1024) = 200KB, therefore I guess at 1280*1024 you
should hit the 160KB barrier very easily (file size of compressed images does
not scale linear of the size of the uncompressed image).

I can provide samples next week on demand (as much you want) ...
The max size is now 500KB.

Gerv
Status: NEW → RESOLVED
Last Resolved: 14 years ago
Resolution: --- → FIXED
Component: Bugzilla: Other b.m.o Issues → General
Product: mozilla.org → bugzilla.mozilla.org
You need to log in before you can comment on or make changes to this bug.