Closed Bug 514083 Opened 15 years ago Closed 7 months ago

Per-file HFS+ compression on Mac OSX 10.6

Categories

(Core :: General, enhancement, P1)

Product:
Core
Component:
General
Platform:
x86
macOS
Type:
enhancement

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: simon.bugzilla, Unassigned)

References

(Blocks 1 open bug,
URL
)

Details

(Whiteboard: [ts])

User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6; en-us) AppleWebKit/532.0+ (KHTML, like Gecko) Version/4.0.3 Safari/531.9 Build Identifier: I was reading this article about Snow Leopard at arstechnica.com, was thinking it could help with [ts]. Reproducible: Always
Whiteboard: [ts]
The link is included, but if missed http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/3
So the suggestion is to drop support for everything older than 10.6, just to get a (presumably small) startup time improvement?
What about a specially optimized Snow Leopard build that uses compression?
Seems like that would mean, at the very least: - User confusion when downloading (anyone pre-10.6 accidentally getting the wrong version would have a mysteriously non-functional download) - Double the QA load for release - Complexity in the build system - Much more complex auto-updating
I have tested Firefox which has been compressed using ditto. When copied to a Leopard machine the file is uncompressed and the compression is not retained, so appears to be backward compatible. As Mozilla 1.9.1+ is Leopard or higher could this be an option? Statistics when compressed. ts needs to be measured. /Applications/Firefox.app: Number of HFS+ compressed files: 372 Total number of files: 404 Total number of folders: 68 Total number of items (number of files + number of folders): 472 Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 51360173 bytes / 52.4 MB (megabytes) / 50 MiB (mebibytes) Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 21597811 bytes / 21.9 MB (megabytes) / 20.9 MiB (mebibytes) Folder size (compressed): 21851141 bytes / 22.2 MB (megabytes) / 21.1 MiB (mebibytes) Compression savings: 57.5% Appoximate total folder size (files + file overhead + folder overhead): 22541904 bytes / 22.5 MB (megabytes) / 21.5 MiB (mebibytes)
Ignore the backward compatible part, my bad. The dmg file I created to copy from one machine to another did not copy over a compressed version.
Simon this looks interesting. I don't have a snow leopard box, Could you check if there is any cold startup improvement from compression? In particular I'm curious about compressing small files that ship with firefox so the file data is stored in the metatada(if i'm reading the article right) Run purge to flush the fs cache and then do https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements#Measuring_Ts
I’ll do some ts testing when I’m at my machine. I’m looking at bug 516362 as an opportunity to implement this. I wonder if we could attach a script to to compress the files after they have been copied, that’s if they are going to choose an pkg installer.
(In reply to comment #8) > I’m looking at bug 516362 as an opportunity to implement this. I wonder if we > could attach a script to to compress the files after they have been copied, > that’s if they are going to choose an pkg installer. First we have to see if this compression thing is a win. If it is we'll likely find a way to make it happen.
Testing was conducted on a MacBook Pro (late 2007) 2.4GHz, fitted with a Western Digital 500GB 5200rpm drive. Purge was run between each test. Latest moz-cen Firefox was used. 10 tests were measured for uncompressed, and 10 more for compressed. Five most closely correlated results were chosen (just from visual inspection). Here we go: Uncompressed 14619 14355 14970 15088 14710 Ave: 14748 Compressed 12568 12295 11938 12060 12667 Ave: 12306 That’s a 16.6% increase in ts when compressed. :)
Joel, can you investigate turning on HFS+ compression from within firefox? (ie on first run?).
Assignee: nobody → joelr
Summary: Per-file HFS+ compression → Per-file HFS+ compression on OSX 10.6
I'm unsure if Apple have a way, other than using ditto, to compress files. brkirch has written a small app that compresses, under GPL v.3 license. afsctool: http://web.me.com/brkirch/brkirchs_Software/afsctool/afsctool.html
Taras, The files need to be pre-compressed. You can't just turn HFS+ compression, you need to compress the files beforehand and, likely, set the compressed flag. Let me know how you want to proceed.
(In reply to comment #13) > Taras, > > The files need to be pre-compressed. You can't just turn HFS+ compression, you > need to compress the files beforehand and, likely, set the compressed flag. > > Let me know how you want to proceed. I was hoping you would a) Test effects of compression on your "static" ff binaries. b) Figure out how to either ship compressed files that work on older OSX platforms OR how to compress files post-installation.
5547, 3852, 4048, 4204, 4008, 4039 - dynamic, not compressed 4177, 3052, 3138, 3112, 3093, 3263 - dynamic, compressed 4746, 3659, 3532, 3446, 3416, 3486 - static, not compressed 4019, 2716, 2849, 2679, 2671, 2769 - static, compressed
Depends on: 525013
It's very simple to compress -any- file, including JARs. Just use ditto --hfsCompression <source> <destination>
At first it didn't seem to make sense to compress jar files. 4790, 3501, 3540, 3397, 3476, 3470 - no pic, compressed, jar compressed 4329, 2876, 2918, 3362, 2819, 2920 - no pic, compressed Then I realized that the jar numbers look awfully like the uncompressed numbers. Why? I duplicated the Minefield.app directory in the Finder and this, apparently, does not preserve compression. I went into the newly copied directory and compressed dynamic libraries and firefox-bin for a nice further speed up! 3923, 2965, 2507, 2592, 2611, 2575 - no pic, compressed, jar compressed Can we break 2s?
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Summary: Per-file HFS+ compression on OSX 10.6 → Per-file HFS+ compression on Mac OSX 10.6
Depends on: 534246
Priority: -- → P1
I think we should get file compression landed. I believe it should be done in the installer and it MUST be done the updater. The installer change is for first-startup experience. The updater needs to do file compression because as files get updated(via copying+mods I presume), files become uncompressed.
Does HFS+ compression still have a performance improvement when used with an SSD disk, or does HFS+ compression even exist for SSD? Or RAID? Does the CPU usage negate the benefits in these higher performance disk setups?
Most if not all Snow Leopard is compressed, this is how Apple was able to reduce size by a few gigabytes. I have the 256Gb Apple SSD and SL flies for me. HFS+ compression does not depend on the hardware and CPUs these days are mostly idle. So yes, it helps and I remember seeing a speed-up from Firefox.
Robert, there will little to no significant speed increase with SSD drives, but there certainly is with slower drives, such as the current 5200RPM 160GB drives that come with today's 13" MacBook Pro machines as an example, that hard drive is 4 years old. The Firefox.app 3.6 RC1 is ~60% smaller in size when compressed, from 52MiB to 20MiB, so you can save valuable hard drive space on your SSD, so you will get some benefit.
Understood. I was really just wondering if it makes sense to use compression on SSD and waste CPU cycles... for example on lower powered laptops. But if Apple is doing it on all Snow Leopard hardware, I'm guessing they did enough tests and found the benefits outweigh the costs.
(In reply to comment #17) > At first it didn't seem to make sense to compress jar files. We currently ship our JAR files without compression. We could simply compress them instead if you'd like to see if that provides a similar win with less effort.
That would e a win for all operating systems, surely? Sounds worth trying to me.
(In reply to comment #23) > (In reply to comment #17) > > At first it didn't seem to make sense to compress jar files. > > We currently ship our JAR files without compression. We could simply compress > them instead if you'd like to see if that provides a similar win with less > effort. I suggested that in bug 524858 as an alternative to minify JS to reduce IO. Seems logical on platforms that aren't mobile (and CPU bound). Do we know what compression algorithm OS X uses? Perhaps playing with different levels or even a diff algorithm (zip vs 7z perhaps) there's an optimal solution that minimizes CPU thrashing and still improves performance.
HFS+ Compression is just zlib, IIRC. Should be the same as zip compression.
Also, this still might be a win on mobile, even with slower CPU, the IO is *much* slower there.
That makes sense. Then it's likely the compression level would be best tuned per platform as mobile devices may benefit from a lower compression level. I'm guessing Apple doesn't use anything higher than 6 though (isn't that the default for most of the web as well?)
Hmm. According to a comment here Apple is using 5: http://www.macosxhints.com/article.php?story=20090902223042255 I'm guessing Apple isn't doing FS compression on the iPhone or it would be evident by now.
(In reply to comment #23) > (In reply to comment #17) > > At first it didn't seem to make sense to compress jar files. > > We currently ship our JAR files without compression. We could simply compress > them instead if you'd like to see if that provides a similar win with less > effort. No, it's a bad idea. I measured it. It's a win for cold startup to use zlib compression, but it hurts warm startup. The benefit of doing it on fs-level is that you read less from disk(and disk is usually slower than cputime for decompression), but once it is in memory, there is no compression overhead.
Should we just run ditto with hfs+ compression enabled to copy ourselves to /Applications?
You mean, we could make an installer that does that? bug 516362
No installer, see bottom of bug 516362. I'm talking about invoking ditto from Firefox to copy ourselves. The question is whether we should come up with the C code to do it or use ditto.
I've tried per-file compression on my complete compiled Thunderbird.app with "$ ditto --hfsCompression". And after that hfsdebug told me for the files inside Thunderbird.app "resource fork has compressed data". But the compressed TB has exactly the same size than the uncompressed one (original: 34,6 MB, compressed: 34,6 MB). How do you compress? And how can I compress while building? The onliest thing I noticed is, the compressed application has a faster startup.
You compressed your TB and hfsdebug confirmed it. Try this to prove it: % ls -lh less -rwxr-xr-x 2 root wheel 285K May 18 11:48 less % du -h -d 0 less 120K less There should be a difference between what ls and du report.
Ah yes, I see a difference. If I use du than I also see a difference between the uncompressed (33 MB) and the compressed (13 MB) one.
And why isn't the compression preserved if I put my application into a DMG? I've experimented with hfscompress my whole obdir and after that create the dmg. But the application in the dmg wasn't compressed anymore. So I've compressed the application in my application folder, I've created a dmg (HFS+J, GUID) and dropped my compressed application into it. Than I closed it, opened it, dropped the application from the dmg to my desktop and analysed it with hfsdebug and du. The application wasn't compressed anymore. (?)
Blocks: 447581
AppleFSCompression is a private framework. Hunting around, there is info on the web at http://www.macosxhints.com/article.php?story=20090902223042255 If you look at the comments, "brkirch" has figured out what is going on, described what he did, and pointed to a GPL program that compresses files like ditto (http://files.me.com/brkirch/ijt4f7)
Christian, see comment 12.
Ah, yep just saw it. Apologies.
Any status update in this Joel? HFS compression has been bought up in bug 516362.
(In reply to comment #42) > Any status update in this Joel? > > HFS compression has been bought up in bug 516362. He isn't working on this anymore.
Assignee: joelr → nobody
No assignee, updating the status.
Status: ASSIGNED → NEW
No assignee, updating the status.
No assignee, updating the status.
No assignee, updating the status.
Severity: normal → S3

i don't think we need it anymore

Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.