[red rocket]

I've recently been encountering a lot of .rar archives, specifically .rar archives concatenated with images. It seems like a pretty good format.

If I want to create a .rar archive on OS X, I either have to use Gumby with the ⌥ key held down or otherwise download and install the rar executable in one of my bin folders.

Now, obviously, rar is proprietary shareware, but the fact remains that it's considerably more straightforward to use on Windows than it is on OS X.

In my opinion, it would be nice if Apple licensed rar. Zip is generally considered the lousiest compression mechanism around, I really don't understand why it's the OS X default.

[mduell]

The zip container is used because it's universal; every OS under the sun supports it and most include it. And it's good enough for reasonably compressible files. DEFLATE is the most common compression algorithm used, although there are a few other options that may work better for your particular type of data.

Zipping, raring, or any other compression of lossy compressed media (images, audio, or video) is stupid. You're not going to get any meaningful compression out of it, and the result may even be a tad bigger due to the overhead of the compression container.

[CatOne]

Zip is pretty good. Rar is typically used for warez (er, "shared videos."). How's that bittorrent client treating you, red rocket red rocket?

[CharlesS]

Originally Posted by red rocket 

Zip is generally considered the lousiest compression mechanism around

The .zip format is just a container, not a compression mechanism. It provides a structure in which to store files, but those files can be compressed using any one of quite a few different methods. It would be easy to implement a version of the .zip format which used bzip2 or LZMA (the compression used in .rar) as its compression method - the only problem there would be compatibility, as those files would only be able to be decompressed by other Zip utilities that supported the same mechanism you implemented. Therefore, most Zip utilities compress using the deflate algorithm, which isn't bad - it's the same algorithm used in gzip - and is supported by pretty much everyone. It's fast, to boot - most of those really tight compression algorithms are also quite slow to compress things.

The one thing about Zip that can make its compression ratio worse than other formats is that it compresses each file separately. In other formats like .tar.gz, which compress the whole archive at once, any redundancies between the files can be accounted for, so for example this text that appears at the beginning of most .plist files:

Code:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">

only needs to be actually stored in the archive once, and the compression can replace the other instances of that text with a reference to the original. The problem with such archive formats is that to get anything out of them, you have to decompress the whole archive. So if you just wanted to extract a 5 KB text file from a 2 GB archive, you'd have to wait for the whole 2 GB to decompress, whereas with Zip, you can get that file out pretty much instantly. So the files won't be quite as small, but they let you do some pretty nice things (just wait until you see the Zip support in Pacifist 2.5 ). Basically, it's a tradeoff between compression ratio and convenience.

[Silky Voice of The Gorn]

The better is XAR.

[CharlesS]

XAR is actually remarkably similar to .zip in how it works. All the files are compressed separately, using either gzip (i.e. the deflate algorithm, which .zip uses), or bzip2. It's very convenient, and you can get individual files out of the archive very quickly, but it won't get the archive as small as just tarring and bzipping it would.

LZMA support could get added to XAR, which would be quite nice. I wouldn't hold my breath for it happening very quickly, though, since I have found the XAR project to be quite slow to incorporate new patches.

[red rocket]

Originally Posted by CatOne

Zip is pretty good. Rar is typically used for warez (er, "shared videos."). How's that bittorrent client treating you, red rocket red rocket?

‘Pretty good’ doesn't mean much. Any comparison between .zip and other formats you can find through Google has .zip right at the bottom, with .7z and .rar near the top.

Rar is bad because people use it for warez? I don't think so. You might want to think about why people use it to compress large files such as videos. One good reason is that you can use .rar to create split archives. Download dozens of individual parts to one location, then have them combined automatically when expanded. That's pretty convenient, regardless of content.

And I don't actually torrent videos. I download a lot of e‑books. Not that I'd read a whole book on the computer, but it's useful for finding out what I like before buying the physical items.

[red rocket]

Originally Posted by CharlesS 

The .zip format is just a container, not a compression mechanism. […] The problem with such archive formats is that to get anything out of them, you have to decompress the whole archive. So if you just wanted to extract a 5 KB text file from a 2 GB archive, you'd have to wait for the whole 2 GB to decompress, whereas with Zip, you can get that file out pretty much instantly. So the files won't be quite as small, but they let you do some pretty nice things (just wait until you see the Zip support in Pacifist 2.5 ). Basically, it's a tradeoff between compression ratio and convenience.

Interesting stuff.

I've got a question. How do you extract individual files from .zip archives at the moment?

[CharlesS]

Originally Posted by red rocket 

Interesting stuff.

I've got a question. How do you extract individual files from .zip archives at the moment?

You wait for Pacifist 2.5 to come out.

Or, if you'd like to help out, you can sign up for the beta test and play with it right now.

Well, there's also a command line tool you can use. I found it to have some problems, though, which is why I decided to write my own Zip support from scratch, hence the reason I could tell you quite a bit about the Zip format now - I had to do a bunch of research to be able to write my Zip parser.

[besson3c]

Let's tackle this differently...

What features do you want out of your archiving/compression tool?

- encryption
- segmentation
- checksumming
- level of compression, if any/smallest possible filesize
- fastest possible performance
- flexible, extensive utility (e.g. being easily incorporated into a Unix based environment)
- open source format/protocol
- maximum compatibility
- handling of Mac metadata

These are some features I can think of. Wouldn't it be prudent to establish which of these features are the most important before recommending a zip replacement? I couldn't map out a table comparing even a fraction of the existing formats and whether or not they have these features, but I'm sure that each format has a slightly different design catering to different needs.

Being a Unixy guy, the following are the most important to me:

- encryption: nice, although with gpg signing and checksumming I could live without this

- segmentation: this would be handy

- checksumming: this seems vital

- smallest possible filesize: I like the idea of separating compression from archiving, because there are times where I don't want to take the time to compress.

- fastest possible performance: don't care so much

- flexible, extensive utility (e.g. being easily incorporated into a Unix based environment): vital

- open source protocol: vital, I'm done with the days of Stuffit and the like. I don't like the idea of a single company controlling the format this way.

- maximum compatibility: hopefully achieved by opening up the format

- handling of Mac metadata: my preference would be for Apple to drop the silly ways it handles metadata and littering of files

[Chuckit]

Maybe I'm misunderstanding, but can't any format be encrypted or checksummed?

I also disagree that metadata is silly.

[besson3c]

Originally Posted by Chuckit 

Maybe I'm misunderstanding, but can't any format be encrypted or checksummed?

I also disagree that metadata is silly.

I didn't say that metadata is silly, it's quite useful, but Apple's handling is silly. They aren't going to come up with a solution that will allow files to be transferred between platforms while leaving this metadata intact without littering files (which creates a new set of problems), so they ought to store the metadata at the file system level.

You can encrypt and checksum any file on your system with a supplementary utility, but I believe xar can actually do this in real-time for you so that there is never a point where the files are unencrypted, to automate the encryption taking place, and to encrypt each individual file in the archive rather than the archive file as a whole...

I could be wrong about some of this though.

[CharlesS]

Originally Posted by besson3c 

I didn't say that metadata is silly, it's quite useful, but Apple's handling is silly. They aren't going to come up with a solution that will allow files to be transferred between platforms while leaving this metadata intact without littering files (which creates a new set of problems), so they ought to store the metadata at the file system level.

Uh, they do. The only time they create an extra file is if the files are sitting on a file system (such as FAT) or an archive format (such as Zip) that doesn't support storing the metadata at the file system level.

[Chuckit]

Originally Posted by CharlesS 

Uh, they do. The only time they create an extra file is if the files are sitting on a file system (such as FAT) or an archive format (such as Zip) that doesn't support storing the metadata at the file system level.

Except for Finder metadata, for whatever reason. I'm pretty sure .DS_Store is still used.

[CharlesS]

Originally Posted by Chuckit 

Except for Finder metadata, for whatever reason. I'm pretty sure .DS_Store is still used.

Yeah, .DS_Store is an exception.

[besson3c]

Originally Posted by CharlesS 

Uh, they do. The only time they create an extra file is if the files are sitting on a file system (such as FAT) or an archive format (such as Zip) that doesn't support storing the metadata at the file system level.

What about .AppleDouble, ._filename, and the files created by an Apple zip archive?

.AppleDouble I believe is the product of AFP, ._filename I'm not certain of (but I see this with my sshfs mounts), and the files created in zip archives we've talked about.

As Chuckit pointed out, .DS_Store is another...

[besson3c]

BTW, just so you are aware CharlesS (I don't mean to center you out, others do this too), starting sentences where you correct somebody with "Uh," can be interpreted as a little condescending. I suppose I may not be the most qualified to talk about being condescending, so please don't take this as criticism... Just pointing this out.

[Chuckit]

Originally Posted by besson3c 

What about .AppleDouble, ._filename, and the files created by an Apple zip archive?

Those would be covered under "files are sitting on a file system (such as FAT) or an archive format (such as Zip) that doesn't support storing the metadata at the file system level," wouldn't they? Those things aren't created on a normal HFS mount.

[TETENAL]

Originally Posted by Chuckit 

Except for Finder metadata, for whatever reason. I'm pretty sure .DS_Store is still used.

The Finder is not a file. It's an application. Therefore there is not "Finder metadata". If you will, what is stored in the .DS_Store files could be described as "folder metadata". Information about how the folder should be displayed. HFS(+) does not natively support storing this information. Therefore having a .DS_Store file in them is not an exception.

[besson3c]

Originally Posted by Chuckit 

Those would be covered under "files are sitting on a file system (such as FAT) or an archive format (such as Zip) that doesn't support storing the metadata at the file system level," wouldn't they? Those things aren't created on a normal HFS mount.

Right, and what I'm saying is that Apple should do away with worrying about transporting files to and from other file systems while preserving Apple metadata. Their whole scheme seems predicated around the idea that you will not disassociate files from their metadata files. What happens if you move your file out of the folder and somewhere else, then send it back to your Mac?

The cost/benefit ratio just seems to work in the favor of forcing Mac users use metadata with things such as Get Info comments to accept the fact that this doesn't work well when sending files to and from non-Macs.

Contrast this to the problems created by all of these files littered about, and my case is made stronger, I think.

What I'm proposing is using xattr. The only system that doesn't support it is Windows, and I haven't even checked to see if this has changed in Vista (probably not).

[TETENAL]

Originally Posted by besson3c 

Apple's handling is silly. They aren't going to come up with a solution that will allow files to be transferred between platforms while leaving this metadata intact without littering files

How do you preserve information while transfering over a channel that can not transport such information without transcribing said information into a supported format (or as you call it "littering")? It's simply not possible.

[besson3c]

Originally Posted by TETENAL 

How do you preserve information while transfering over a channel that can not transport such information without transcribing said information into a supported format (or as you call it "littering")? It's simply not possible.

You use xattr, and if the other system doesn't support it, tough... Like I said, Apple's scheme only works when you don't disassociate the files from their metadata anyway. Any file that is uploaded to a web server from a Windows machine, emailed somewhere, or whatever will not result in the associated metadata files making that trip.

[Chuckit]

Originally Posted by TETENAL 

The Finder is not a file. It's an application. Therefore there is not "Finder metadata".

That's a fairly nonsensical comment. It's metadata that's primarily used by the Finder — hence, Finder metadata.

Originally Posted by TETENAL 

If you will, what is stored in the .DS_Store files could be described as "folder metadata". Information about how the folder should be displayed. HFS(+) does not natively support storing this information.

It could be stored as xattr data just as well as anything else. There's nothing about HFS+ that forces you to store comments or labels on a file as a completely separate data file.

[Chuckit]

Originally Posted by besson3c 

The cost/benefit ratio just seems to work in the favor of forcing Mac users use metadata with things such as Get Info comments to accept the fact that this doesn't work well when sending files to and from non-Macs.

Contrast this to the problems created by all of these files littered about, and my case is made stronger, I think.

I really think not. Which would you rather have:
• 10,000 harmless, invisible, essentially empty files scattered around your hard drive
• Half your database randomly erased and lost from your backups

Unintended data loss is the worst outcome possible in computing. Extra files are annoying, I'm sure, but they are not destructive. I have all sorts of useless **** that came with Unix OSS I've downloaded and compiled, but I'm not crying about how the big, mean Unix people are causing me problems with all their files.

[besson3c]

Originally Posted by Chuckit 

I really think not. Which would you rather have:
• 10,000 harmless, invisible, essentially empty files scattered around your hard drive
• Half your database randomly erased and lost from your backups

Unintended data loss is the worst outcome possible in computing. Extra files are annoying, I'm sure, but they are not destructive. I have all sorts of useless **** that came with Unix OSS I've downloaded and compiled, but I'm not crying about how the big, mean Unix people are causing me problems with all their files.

What do you think some of the problems of having these extra files are?

Exactly what data is stored as Finder metadata? Labels, Get Info comments, window position, perhaps whether or not the extension will be displayed? What else? Changing behavior to not rely on Get Info comments and Labels seems like a far better solution than some of the potential security problems, extra inode usage, more file writes, slower file searches, etc.

[CharlesS]

Originally Posted by besson3c 

Right, and what I'm saying is that Apple should do away with worrying about transporting files to and from other file systems while preserving Apple metadata. Their whole scheme seems predicated around the idea that you will not disassociate files from their metadata files. What happens if you move your file out of the folder and somewhere else, then send it back to your Mac?

Clearly something different from if you had just nuked the metadata on any file that was stored in a Zip archive or on a FAT filesystem, right?

Actually the system is designed to keep stuff from getting damaged when being transferred from Mac to Mac. For example, it is now common for Mac users to send files to each other inside .zip archives. Therefore, there needs to be a way to preserve Mac information in those .zip archives. Similarly, it is common for Mac users to transfer files on USB flash drives, which are almost always pre-formatted to the FAT filesystem. If your user's file gets nuked just because he had the misfortune to put it on a USB flash drive, that user is generally not going to be happy with you.

[besson3c]

Charles and Chuckit,

I can see your point here. It seems prudent that for those that care about preserving Finder metadata over the problems associated with littering files, this should be an option, if not the default. I also think that there should be the option *somewhere* to disable the littering too. This actually does create some serious security related problems for some people, it is more than simply annoying.

[CharlesS]

Well, I can't really argue with you there (except that I do feel that preserving the metadata should be the default setting).

[miawj]

Never mind extracting a single file. It was, embarassingly, literally years before I worked out how to uncompress a whole archive. I worked out you could use the ctrl key to to compress folders but no suitable option popped up when I tried to reverse the process. I was dragging and dropping to 'StuffIt Expander' or 'The Unarchiver' for ages before I noticed the results of double-clicking an archive.

[Kevin]

Originally Posted by besson3c 

- handling of Mac metadata: my preference would be for Apple to drop the silly ways it handles metadata and littering of files

I think you can find a way to interject a bash at Apple for not doing things "your way" in about any thread.

I wonder if any Apple users go into Unix forums and do this.

Originally Posted by besson3c 

I suppose I may not be the most qualified to talk about being condescending,

Nevermind.

[besson3c]

Kevin, I just wish you could find a way to ignore me. Please keep on trying. Thanks!