Last Comment Bug 514083 - Per-file HFS+ compression on Mac OSX 10.6
: Per-file HFS+ compression on Mac OSX 10.6
Status: ASSIGNED
[ts]
:
Product: Core
Classification: Components
Component: General (show other bugs)
: unspecified
: x86 Mac OS X
: P1 enhancement with 9 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
http://arstechnica.com/apple/reviews/...
Depends on: 534246 525013
Blocks: 447581
  Show dependency treegraph
 
Reported: 2009-09-01 16:59 PDT by Simon Howes
Modified: 2013-01-09 14:15 PST (History)
20 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments

Description Simon Howes 2009-09-01 16:59:46 PDT
User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6; en-us) AppleWebKit/532.0+ (KHTML, like Gecko) Version/4.0.3 Safari/531.9
Build Identifier: 

I was reading this article about Snow Leopard at arstechnica.com, was thinking it could help with [ts].

Reproducible: Always
Comment 1 Simon Howes 2009-09-01 17:05:24 PDT
The link is included, but if missed http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/3
Comment 2 Stuart Morgan 2009-09-11 15:05:29 PDT
So the suggestion is to drop support for everything older than 10.6, just to get a (presumably small) startup time improvement?
Comment 3 Joel Reymont (:joelr) 2009-09-11 15:07:16 PDT
What about a specially optimized Snow Leopard build that uses compression?
Comment 4 Stuart Morgan 2009-09-11 16:16:19 PDT
Seems like that would mean, at the very least:
- User confusion when downloading (anyone pre-10.6 accidentally getting the wrong version would have a mysteriously non-functional download)
- Double the QA load for release
- Complexity in the build system
- Much more complex auto-updating
Comment 5 Simon Howes 2009-09-13 07:35:46 PDT
I have tested Firefox which has been compressed using ditto.  When copied to a Leopard machine the file is uncompressed and the compression is not retained, so appears to be backward compatible.  As Mozilla 1.9.1+ is Leopard or higher could this be an option?


Statistics when compressed.  ts needs to be measured.

/Applications/Firefox.app:
Number of HFS+ compressed files: 372
Total number of files: 404
Total number of folders: 68
Total number of items (number of files + number of folders): 472
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 51360173 bytes / 52.4 MB (megabytes) / 50 MiB (mebibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 21597811 bytes / 21.9 MB (megabytes) / 20.9 MiB (mebibytes)
Folder size (compressed): 21851141 bytes / 22.2 MB (megabytes) / 21.1 MiB (mebibytes)
Compression savings: 57.5%
Appoximate total folder size (files + file overhead + folder overhead): 22541904 bytes / 22.5 MB (megabytes) / 21.5 MiB (mebibytes)
Comment 6 Simon Howes 2009-09-13 07:51:23 PDT
Ignore the backward compatible part, my bad. The dmg file I created to copy from one machine to another did not copy over a compressed version.
Comment 7 (dormant account) 2009-10-14 14:30:54 PDT
Simon this looks interesting. I don't have a snow leopard box, Could you check if there is any cold startup improvement from compression? In particular I'm curious about compressing small files that ship with firefox so the file data is stored in the metatada(if i'm reading the article right)


Run purge to flush the fs cache and then do https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements#Measuring_Ts
Comment 8 Simon Howes 2009-10-14 14:42:59 PDT
I’ll do some ts testing when I’m at my machine.

I’m looking at bug 516362 as an opportunity to implement this. I wonder if we could attach a script to  to compress the files after they have been copied, that’s if they are going to choose an pkg installer.
Comment 9 (dormant account) 2009-10-14 14:46:00 PDT
(In reply to comment #8)

> I’m looking at bug 516362 as an opportunity to implement this. I wonder if we
> could attach a script to  to compress the files after they have been copied,
> that’s if they are going to choose an pkg installer.

First we have to see if this compression thing is a win. If it is we'll likely find a way to make it happen.
Comment 10 Simon Howes 2009-10-14 15:39:51 PDT
Testing was conducted on a MacBook Pro (late 2007) 2.4GHz, fitted with a Western Digital 500GB 5200rpm drive.

Purge was run between each test. Latest moz-cen Firefox was used. 10 tests were measured for uncompressed, and 10 more for compressed. Five most closely correlated results were chosen (just from visual inspection). 

Here we go:

Uncompressed
14619
14355
14970
15088
14710

Ave: 14748


Compressed
12568
12295
11938
12060
12667

Ave: 12306

That’s a 16.6% increase in ts when compressed. :)
Comment 11 (dormant account) 2009-12-08 17:24:34 PST
Joel, can you investigate turning on HFS+ compression from within firefox? (ie on first run?).
Comment 12 Simon Howes 2009-12-09 04:35:27 PST
I'm unsure if Apple have a way, other than using ditto, to compress files. brkirch has written a small app that compresses, under GPL v.3 license.

afsctool: http://web.me.com/brkirch/brkirchs_Software/afsctool/afsctool.html
Comment 13 Joel Reymont (:joelr) 2009-12-09 05:24:04 PST
Taras, 

The files need to be pre-compressed. You can't just turn HFS+ compression, you need to compress the files beforehand and, likely, set the compressed flag. 

Let me know how you want to proceed.
Comment 14 (dormant account) 2009-12-09 08:55:17 PST
(In reply to comment #13)
> Taras, 
> 
> The files need to be pre-compressed. You can't just turn HFS+ compression, you
> need to compress the files beforehand and, likely, set the compressed flag. 
> 
> Let me know how you want to proceed.

I was hoping you would
a) Test effects of compression on your "static" ff binaries.
b) Figure out how to either ship compressed files that work on older OSX platforms OR how to compress files post-installation.
Comment 15 Joel Reymont (:joelr) 2009-12-09 15:09:32 PST
5547, 3852, 4048, 4204, 4008, 4039 - dynamic, not compressed
4177, 3052, 3138, 3112, 3093, 3263 - dynamic, compressed

4746, 3659, 3532, 3446, 3416, 3486 - static, not compressed
4019, 2716, 2849, 2679, 2671, 2769 - static, compressed
Comment 16 Joel Reymont (:joelr) 2009-12-09 15:52:35 PST
It's very simple to compress -any- file, including JARs. Just use

ditto --hfsCompression <source> <destination>
Comment 17 Joel Reymont (:joelr) 2009-12-10 06:21:39 PST
At first it didn't seem to make sense to compress jar files.

4790, 3501, 3540, 3397, 3476, 3470 - no pic, compressed, jar compressed
4329, 2876, 2918, 3362, 2819, 2920 - no pic, compressed

Then I realized that the jar numbers look awfully like the uncompressed numbers. Why? I duplicated the Minefield.app directory in the Finder and this, apparently, does not preserve compression. 

I went into the newly copied directory and compressed dynamic libraries and firefox-bin for a nice further speed up!

3923, 2965, 2507, 2592, 2611, 2575 - no pic, compressed, jar compressed

Can we break 2s?
Comment 18 (dormant account) 2010-01-13 14:46:31 PST
I think we should get file compression landed. I believe it should be done in the installer and it MUST be done the updater.

The installer change is for first-startup experience. The updater needs to do file compression because as files get updated(via copying+mods I presume), files become uncompressed.
Comment 19 Robert Accettura [:raccettura] 2010-01-13 19:11:49 PST
Does HFS+ compression still have a performance improvement when used with an SSD disk, or does HFS+ compression even exist for SSD?  Or RAID?  Does the CPU usage negate the benefits in these higher performance disk setups?
Comment 20 Joel Reymont (:joelr) 2010-01-13 23:17:02 PST
Most if not all Snow Leopard is compressed, this is how Apple was able to reduce size by a few gigabytes. I have the 256Gb Apple SSD and SL flies for me. 

HFS+ compression does not depend on the hardware and CPUs these days are mostly idle. So yes, it helps and I remember seeing a speed-up from Firefox.
Comment 21 Simon Howes 2010-01-14 05:21:40 PST
Robert, there will little to no significant speed increase with SSD drives, but there certainly is with slower drives, such as the current 5200RPM 160GB drives that come with today's 13" MacBook Pro machines as an example, that hard drive is 4 years old. The Firefox.app 3.6 RC1 is ~60% smaller in size when compressed, from 52MiB to 20MiB, so you can save valuable hard drive space on your SSD, so you will get some benefit.
Comment 22 Robert Accettura [:raccettura] 2010-01-14 18:39:51 PST
Understood.  I was really just wondering if it makes sense to use compression on SSD and waste CPU cycles... for example on lower powered laptops.  But if Apple is doing it on all Snow Leopard hardware, I'm guessing they did enough tests and found the benefits outweigh the costs.
Comment 23 Ted Mielczarek [:ted.mielczarek] 2010-01-17 06:07:46 PST
(In reply to comment #17)
> At first it didn't seem to make sense to compress jar files.

We currently ship our JAR files without compression. We could simply compress them instead if you'd like to see if that provides a similar win with less effort.
Comment 24 Simon Howes 2010-01-17 06:51:54 PST
That would e a win for all operating systems, surely? Sounds worth trying to me.
Comment 25 Robert Accettura [:raccettura] 2010-01-17 09:55:10 PST
(In reply to comment #23)
> (In reply to comment #17)
> > At first it didn't seem to make sense to compress jar files.
> 
> We currently ship our JAR files without compression. We could simply compress
> them instead if you'd like to see if that provides a similar win with less
> effort.

I suggested that in bug 524858 as an alternative to minify JS to reduce IO.  Seems logical on platforms that aren't mobile (and CPU bound).

Do we know what compression algorithm OS X uses?  Perhaps playing with different levels or even a diff algorithm (zip vs 7z perhaps) there's an optimal solution that minimizes CPU thrashing and still improves performance.
Comment 26 Ted Mielczarek [:ted.mielczarek] 2010-01-17 10:39:17 PST
HFS+ Compression is just zlib, IIRC. Should be the same as zip compression.
Comment 27 Ted Mielczarek [:ted.mielczarek] 2010-01-17 10:39:38 PST
Also, this still might be a win on mobile, even with slower CPU, the IO is *much* slower there.
Comment 28 Robert Accettura [:raccettura] 2010-01-17 12:02:46 PST
That makes sense.  Then it's likely the compression level would be best tuned per platform as mobile devices may benefit from a lower compression level.  I'm guessing Apple doesn't use anything higher than 6 though (isn't that the default for most of the web as well?)
Comment 29 Robert Accettura [:raccettura] 2010-01-17 12:06:21 PST
Hmm.  According to a comment here Apple is using 5:
http://www.macosxhints.com/article.php?story=20090902223042255

I'm guessing Apple isn't doing FS compression on the iPhone or it would be evident by now.
Comment 30 (dormant account) 2010-01-19 12:32:57 PST
(In reply to comment #23)
> (In reply to comment #17)
> > At first it didn't seem to make sense to compress jar files.
> 
> We currently ship our JAR files without compression. We could simply compress
> them instead if you'd like to see if that provides a similar win with less
> effort.

No, it's a bad idea. I measured it. It's a win for cold startup to use zlib compression, but it hurts warm startup. The benefit of doing it on fs-level is that you read less from disk(and disk is usually slower than cputime for decompression), but once it is in memory, there is no compression overhead.
Comment 31 Joel Reymont (:joelr) 2010-01-26 12:30:43 PST
Should we just run ditto with hfs+ compression enabled to copy ourselves to /Applications?
Comment 32 Simon Howes 2010-01-26 14:06:52 PST
You mean, we could make an installer that does that? bug 516362
Comment 33 Joel Reymont (:joelr) 2010-01-26 14:25:21 PST
No installer, see bottom of bug 516362. I'm talking about invoking ditto from Firefox to copy ourselves. The question is whether we should come up with the C code to do it or use ditto.
Comment 34 Nomis101 2010-02-03 16:42:09 PST
I've tried per-file compression on my complete compiled Thunderbird.app with "$ ditto --hfsCompression". And after that hfsdebug told me for the files inside Thunderbird.app "resource fork has compressed data". But the compressed TB has exactly the same size than the uncompressed one (original: 34,6 MB, compressed: 34,6 MB). How do you compress? And how can I compress while building? The onliest thing I noticed is, the compressed application has a faster startup.
Comment 35 Joel Reymont (:joelr) 2010-02-03 23:18:45 PST
You compressed your TB and hfsdebug confirmed it. 

Try this to prove it:

% ls -lh less
-rwxr-xr-x  2 root  wheel   285K May 18 11:48 less

% du -h -d 0 less           120K    less
 
There should be a difference between what ls and du report.
Comment 36 Nomis101 2010-02-04 02:27:58 PST
Ah yes, I see a difference. If I use du than I also see a difference between the uncompressed (33 MB) and the compressed (13 MB) one.
Comment 37 Nomis101 2010-02-04 05:07:47 PST
And why isn't the compression preserved if I put my application into a DMG? I've experimented with hfscompress my whole obdir and after that create the dmg. But the application in the dmg wasn't compressed anymore. So I've compressed the application in my application folder, I've created a dmg (HFS+J, GUID) and dropped my compressed application into it. Than I closed it, opened it, dropped the application from the dmg to my desktop and analysed it with hfsdebug and du. The application wasn't compressed anymore. (?)
Comment 38 Joel Reymont (:joelr) 2010-02-04 05:12:02 PST
The right place to ask:

http://lists.apple.com/mailman/listinfo/darwin-dev
Comment 39 christian 2010-04-12 17:21:22 PDT
AppleFSCompression is a private framework. Hunting around, there is info on the web at http://www.macosxhints.com/article.php?story=20090902223042255

If you look at the comments, "brkirch" has figured out what is going on, described what he did, and pointed to a GPL program that compresses files like ditto (http://files.me.com/brkirch/ijt4f7)
Comment 40 Simon Howes 2010-04-12 17:30:05 PDT
Christian, see comment 12.
Comment 41 christian 2010-04-12 17:43:26 PDT
Ah, yep just saw it. Apologies.
Comment 42 Simon Howes 2010-09-08 14:59:19 PDT
Any status update in this Joel?

HFS compression has been bought up in bug 516362.
Comment 43 (dormant account) 2010-09-08 15:08:13 PDT
(In reply to comment #42)
> Any status update in this Joel?
> 
> HFS compression has been bought up in bug 516362.

He isn't working on this anymore.

Note You need to log in before you can comment on or make changes to this bug.