Per-file HFS+ compression on Mac OSX 10.6

ASSIGNED
Unassigned

Status

()

Core
General
P1
enhancement
ASSIGNED
8 years ago
4 years ago

People

(Reporter: Simon Howes, Unassigned)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [ts], URL)

(Reporter)

Description

8 years ago
User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6; en-us) AppleWebKit/532.0+ (KHTML, like Gecko) Version/4.0.3 Safari/531.9
Build Identifier: 

I was reading this article about Snow Leopard at arstechnica.com, was thinking it could help with [ts].

Reproducible: Always
(Reporter)

Updated

8 years ago
Whiteboard: [ts]
(Reporter)

Comment 1

8 years ago
The link is included, but if missed http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/3

Comment 2

8 years ago
So the suggestion is to drop support for everything older than 10.6, just to get a (presumably small) startup time improvement?
What about a specially optimized Snow Leopard build that uses compression?

Comment 4

8 years ago
Seems like that would mean, at the very least:
- User confusion when downloading (anyone pre-10.6 accidentally getting the wrong version would have a mysteriously non-functional download)
- Double the QA load for release
- Complexity in the build system
- Much more complex auto-updating
(Reporter)

Comment 5

8 years ago
I have tested Firefox which has been compressed using ditto.  When copied to a Leopard machine the file is uncompressed and the compression is not retained, so appears to be backward compatible.  As Mozilla 1.9.1+ is Leopard or higher could this be an option?


Statistics when compressed.  ts needs to be measured.

/Applications/Firefox.app:
Number of HFS+ compressed files: 372
Total number of files: 404
Total number of folders: 68
Total number of items (number of files + number of folders): 472
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 51360173 bytes / 52.4 MB (megabytes) / 50 MiB (mebibytes)
Folder size (compressed - decmpfs xattr; reported size by Mac OS 10.0-10.5 Finder): 21597811 bytes / 21.9 MB (megabytes) / 20.9 MiB (mebibytes)
Folder size (compressed): 21851141 bytes / 22.2 MB (megabytes) / 21.1 MiB (mebibytes)
Compression savings: 57.5%
Appoximate total folder size (files + file overhead + folder overhead): 22541904 bytes / 22.5 MB (megabytes) / 21.5 MiB (mebibytes)
(Reporter)

Comment 6

8 years ago
Ignore the backward compatible part, my bad. The dmg file I created to copy from one machine to another did not copy over a compressed version.

Comment 7

8 years ago
Simon this looks interesting. I don't have a snow leopard box, Could you check if there is any cold startup improvement from compression? In particular I'm curious about compressing small files that ship with firefox so the file data is stored in the metatada(if i'm reading the article right)


Run purge to flush the fs cache and then do https://wiki.mozilla.org/Firefox/Sprints/Startup_Time_Improvements#Measuring_Ts
(Reporter)

Comment 8

8 years ago
I’ll do some ts testing when I’m at my machine.

I’m looking at bug 516362 as an opportunity to implement this. I wonder if we could attach a script to  to compress the files after they have been copied, that’s if they are going to choose an pkg installer.

Comment 9

8 years ago
(In reply to comment #8)

> I’m looking at bug 516362 as an opportunity to implement this. I wonder if we
> could attach a script to  to compress the files after they have been copied,
> that’s if they are going to choose an pkg installer.

First we have to see if this compression thing is a win. If it is we'll likely find a way to make it happen.
(Reporter)

Comment 10

8 years ago
Testing was conducted on a MacBook Pro (late 2007) 2.4GHz, fitted with a Western Digital 500GB 5200rpm drive.

Purge was run between each test. Latest moz-cen Firefox was used. 10 tests were measured for uncompressed, and 10 more for compressed. Five most closely correlated results were chosen (just from visual inspection). 

Here we go:

Uncompressed
14619
14355
14970
15088
14710

Ave: 14748


Compressed
12568
12295
11938
12060
12667

Ave: 12306

That’s a 16.6% increase in ts when compressed. :)

Comment 11

8 years ago
Joel, can you investigate turning on HFS+ compression from within firefox? (ie on first run?).
Assignee: nobody → joelr

Updated

8 years ago
Summary: Per-file HFS+ compression → Per-file HFS+ compression on OSX 10.6
(Reporter)

Comment 12

8 years ago
I'm unsure if Apple have a way, other than using ditto, to compress files. brkirch has written a small app that compresses, under GPL v.3 license.

afsctool: http://web.me.com/brkirch/brkirchs_Software/afsctool/afsctool.html
Taras, 

The files need to be pre-compressed. You can't just turn HFS+ compression, you need to compress the files beforehand and, likely, set the compressed flag. 

Let me know how you want to proceed.

Comment 14

8 years ago
(In reply to comment #13)
> Taras, 
> 
> The files need to be pre-compressed. You can't just turn HFS+ compression, you
> need to compress the files beforehand and, likely, set the compressed flag. 
> 
> Let me know how you want to proceed.

I was hoping you would
a) Test effects of compression on your "static" ff binaries.
b) Figure out how to either ship compressed files that work on older OSX platforms OR how to compress files post-installation.
5547, 3852, 4048, 4204, 4008, 4039 - dynamic, not compressed
4177, 3052, 3138, 3112, 3093, 3263 - dynamic, compressed

4746, 3659, 3532, 3446, 3416, 3486 - static, not compressed
4019, 2716, 2849, 2679, 2671, 2769 - static, compressed

Updated

8 years ago
Depends on: 525013
It's very simple to compress -any- file, including JARs. Just use

ditto --hfsCompression <source> <destination>
At first it didn't seem to make sense to compress jar files.

4790, 3501, 3540, 3397, 3476, 3470 - no pic, compressed, jar compressed
4329, 2876, 2918, 3362, 2819, 2920 - no pic, compressed

Then I realized that the jar numbers look awfully like the uncompressed numbers. Why? I duplicated the Minefield.app directory in the Finder and this, apparently, does not preserve compression. 

I went into the newly copied directory and compressed dynamic libraries and firefox-bin for a nice further speed up!

3923, 2965, 2507, 2592, 2611, 2575 - no pic, compressed, jar compressed

Can we break 2s?
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Updated

8 years ago
Summary: Per-file HFS+ compression on OSX 10.6 → Per-file HFS+ compression on Mac OSX 10.6

Updated

8 years ago
Depends on: 534246
Priority: -- → P1

Comment 18

7 years ago
I think we should get file compression landed. I believe it should be done in the installer and it MUST be done the updater.

The installer change is for first-startup experience. The updater needs to do file compression because as files get updated(via copying+mods I presume), files become uncompressed.
Does HFS+ compression still have a performance improvement when used with an SSD disk, or does HFS+ compression even exist for SSD?  Or RAID?  Does the CPU usage negate the benefits in these higher performance disk setups?
Most if not all Snow Leopard is compressed, this is how Apple was able to reduce size by a few gigabytes. I have the 256Gb Apple SSD and SL flies for me. 

HFS+ compression does not depend on the hardware and CPUs these days are mostly idle. So yes, it helps and I remember seeing a speed-up from Firefox.
(Reporter)

Comment 21

7 years ago
Robert, there will little to no significant speed increase with SSD drives, but there certainly is with slower drives, such as the current 5200RPM 160GB drives that come with today's 13" MacBook Pro machines as an example, that hard drive is 4 years old. The Firefox.app 3.6 RC1 is ~60% smaller in size when compressed, from 52MiB to 20MiB, so you can save valuable hard drive space on your SSD, so you will get some benefit.
Understood.  I was really just wondering if it makes sense to use compression on SSD and waste CPU cycles... for example on lower powered laptops.  But if Apple is doing it on all Snow Leopard hardware, I'm guessing they did enough tests and found the benefits outweigh the costs.
(In reply to comment #17)
> At first it didn't seem to make sense to compress jar files.

We currently ship our JAR files without compression. We could simply compress them instead if you'd like to see if that provides a similar win with less effort.
(Reporter)

Comment 24

7 years ago
That would e a win for all operating systems, surely? Sounds worth trying to me.
(In reply to comment #23)
> (In reply to comment #17)
> > At first it didn't seem to make sense to compress jar files.
> 
> We currently ship our JAR files without compression. We could simply compress
> them instead if you'd like to see if that provides a similar win with less
> effort.

I suggested that in bug 524858 as an alternative to minify JS to reduce IO.  Seems logical on platforms that aren't mobile (and CPU bound).

Do we know what compression algorithm OS X uses?  Perhaps playing with different levels or even a diff algorithm (zip vs 7z perhaps) there's an optimal solution that minimizes CPU thrashing and still improves performance.
HFS+ Compression is just zlib, IIRC. Should be the same as zip compression.
Also, this still might be a win on mobile, even with slower CPU, the IO is *much* slower there.
That makes sense.  Then it's likely the compression level would be best tuned per platform as mobile devices may benefit from a lower compression level.  I'm guessing Apple doesn't use anything higher than 6 though (isn't that the default for most of the web as well?)
Hmm.  According to a comment here Apple is using 5:
http://www.macosxhints.com/article.php?story=20090902223042255

I'm guessing Apple isn't doing FS compression on the iPhone or it would be evident by now.

Comment 30

7 years ago
(In reply to comment #23)
> (In reply to comment #17)
> > At first it didn't seem to make sense to compress jar files.
> 
> We currently ship our JAR files without compression. We could simply compress
> them instead if you'd like to see if that provides a similar win with less
> effort.

No, it's a bad idea. I measured it. It's a win for cold startup to use zlib compression, but it hurts warm startup. The benefit of doing it on fs-level is that you read less from disk(and disk is usually slower than cputime for decompression), but once it is in memory, there is no compression overhead.
Should we just run ditto with hfs+ compression enabled to copy ourselves to /Applications?
(Reporter)

Comment 32

7 years ago
You mean, we could make an installer that does that? bug 516362
No installer, see bottom of bug 516362. I'm talking about invoking ditto from Firefox to copy ourselves. The question is whether we should come up with the C code to do it or use ditto.

Comment 34

7 years ago
I've tried per-file compression on my complete compiled Thunderbird.app with "$ ditto --hfsCompression". And after that hfsdebug told me for the files inside Thunderbird.app "resource fork has compressed data". But the compressed TB has exactly the same size than the uncompressed one (original: 34,6 MB, compressed: 34,6 MB). How do you compress? And how can I compress while building? The onliest thing I noticed is, the compressed application has a faster startup.
You compressed your TB and hfsdebug confirmed it. 

Try this to prove it:

% ls -lh less
-rwxr-xr-x  2 root  wheel   285K May 18 11:48 less

% du -h -d 0 less           120K    less
 
There should be a difference between what ls and du report.

Comment 36

7 years ago
Ah yes, I see a difference. If I use du than I also see a difference between the uncompressed (33 MB) and the compressed (13 MB) one.

Comment 37

7 years ago
And why isn't the compression preserved if I put my application into a DMG? I've experimented with hfscompress my whole obdir and after that create the dmg. But the application in the dmg wasn't compressed anymore. So I've compressed the application in my application folder, I've created a dmg (HFS+J, GUID) and dropped my compressed application into it. Than I closed it, opened it, dropped the application from the dmg to my desktop and analysed it with hfsdebug and du. The application wasn't compressed anymore. (?)
The right place to ask:

http://lists.apple.com/mailman/listinfo/darwin-dev
Blocks: 447581

Comment 39

7 years ago
AppleFSCompression is a private framework. Hunting around, there is info on the web at http://www.macosxhints.com/article.php?story=20090902223042255

If you look at the comments, "brkirch" has figured out what is going on, described what he did, and pointed to a GPL program that compresses files like ditto (http://files.me.com/brkirch/ijt4f7)
(Reporter)

Comment 40

7 years ago
Christian, see comment 12.

Comment 41

7 years ago
Ah, yep just saw it. Apologies.
(Reporter)

Comment 42

7 years ago
Any status update in this Joel?

HFS compression has been bought up in bug 516362.

Comment 43

7 years ago
(In reply to comment #42)
> Any status update in this Joel?
> 
> HFS compression has been bought up in bug 516362.

He isn't working on this anymore.
Assignee: joelr → nobody
You need to log in before you can comment on or make changes to this bug.