[Eug]

I have a 11 GB Parallels hard drive OS installation file which I zipped up to 7 GB with OS X, for a backup. Are there any compatibility issues?

Also, I thought that some zip versions had a limitation of 4 GB. Even if there is no such 4 GB limitation in OS X, are there any file size limitations in OS X?

Am I better or worse off dealing with dmg files?

[CharlesS]

Does that 7 GB zip file actually work? I'd double-check it by extracting and making sure everything's there. I had to read up on the zip format in order to add support for it in Pacifist, and it uses 32-bit integers for all file sizes, so it really shouldn't work for more than 4 GB.

[Eug]

Originally Posted by CharlesS 

it uses 32-bit integers for all file sizes, so it really shouldn't work for more than 4 GB.

Hmmm... Even in Leopard? I've already started burning it to DVD (dual-layer), but I'll give the file a shot after it's finished recording.

However, maybe I should just scrap it just in case, and use a dmg file. I do know though that there are 64-bit versions of the zip format, and because my zip appeared to work, I just assumed Leopard was using a 64-bit version or whatever.

[CharlesS]

For normal-sized files, it definitely uses the regular 32-bit version of zip. I suppose it is possible that it could be smart enough to switch to Zip64 if it detects that you're compressing a sufficiently large amount of data. I'll have to try this myself to see what it does.

Regardless, I think a disk image is probably the safest way to go in any case.

[Eug]

I took the file to my MacBook, and it unzipped just fine.

Then I tried it on Windows, and the built-in WIndows unzipper says the zip file is corrupted. I also tried FZ Zip, and it just launches (but doesn't actually unzip it) and shuts down immediately. Both programs work fine with a smaller (13 MB) file.

I guess Mac OS X's built-in zipper uses 32-bit for most files, but for über large files (presumably) switches to 64-bit (which is incompatible with most unzippers out there).

[CharlesS]

Hmm, interesting. I didn't expect them to use Zip64 since, as you say, almost no-one supports it. Now in order to be fully compatible I have to go implement Zip64 in Pacifist. Thanks a lot.

[Eug]

P.S. This 11 GB Parallels .hdd file is actually a package. The unzipped file maintained the package structure. When I peeked inside, the package components were intact.

[Big Mac]

I have gotten one complaint that a zip file I sent was corrupted, but I cannot tell if that was due to corruption or Mac encoding.

[CharlesS]

Update: I've been looking at some large zip files created by the Apple archive utility, and they do not appear to be Zip64 archives, AFAICT. They appear to be regular 32-bit Zip archives, with the higher 32 bits being chopped off of every 64-bit integer - for example, 0x123456789 becomes 0x23456789. To be honest, I have no idea how Apple's utility is unarchiving the files correctly - they're probably doing something like ignoring all the offsets and file sizes and just reading all the data in order, detecting when zipped data shows up and where it ends. From what I'm looking at, it appears these really are corrupt archives, and I wouldn't recommend using them for amounts of data larger than 4 GB.

[Eug]

What? Weird. I didn't even know that was possible. I assume your zipped and unzipped >4 GB files worked fine?

Anyways, I guess I'll re-backup this in a dmg just to be safe. Ironically, checking back at my old backups, they're all in dmgs anyway. I previously had just assumed >4 GB zips wouldn't work and didn't even try until this time around. (The only reason I tried this time was because I totally forgot about the size issue until after I had zipped it.)

[CharlesS]

Originally Posted by Eug 

What? Weird. I didn't even know that was possible. I assume your zipped and unzipped >4 GB files worked fine?

It's weird all right. I didn't test the files extensively, but they were all there and seemed to have the right sizes.

The thing about Zip archives is that they store the file information twice. At the beginning they have all the compressed files stored in order, preceded by metadata such as their filename, sizes, etc. At the end of the file, there's a directory which contains all the file metadata again, but not the files' actual compressed data, with an offset that says where to find the actual file in the archive. An app like Pacifist that wants to list the contents of the file before compressing it would read the directory at the end, and then when you wanted to extract something, it looks up its offset and pulls out the file from where it sits in the earlier part of the archive. Typically you'd then use the compressed size info to determine how much data following the record constitutes the file data you're trying to extract. So something like what Apple's archiver is doing with large files will completely mess up Pacifist, which won't know where to look anything up because the offsets are wrong, but Apple's archiver is probably ignoring all that and just starting from the beginning and grabbing all the compressed data it sees. Presumably, it must be ignoring the file sizes too, and just analyzing the compressed data itself to figure out where it ends (which I think ZLib is able to do for you, although some other compressors can't). I probably wouldn't rely on this, because if the behavior of Apple's archive utility changes, you won't be able to get anything out of the archive anymore, and it seems most third-party utilities aren't able to do it either.

As it turns out, though, version 3.0 of Info-Zip (available at http://www.info-zip.org) does automatically make proper Zip64 archives if the file size gets over 4 GB, so if Apple ends up upgrading the included version of Info-Zip to 3.0 in Snow Leopard, they might end up getting this for free - so I'm going to preemptively add support for Zip64 to Pacifist anyway. And of course if you're interested in making big zip files, you can download Zip 3.0 and make them at the command line.