[CharlesS]

I'm seeing an issue in Time Machine in Mavericks. Basically the issue came up when I bought a new MacBook Pro, and tried to migrate my files from my old laptop on it. Since the Thunderbolt->FireWire adapter I ordered was late getting here, I simply updated the TM backup on my old MBP, and then migrated the new MBP from the TM backup. Immediately upon starting up, it became evident that something was wrong; many of my apps' settings were reset to the defaults, shareware apps that were previously registered were now prompting for a regcode, etc. What's worse was that some apps weren't working at all. Further investigation showed that certain files had not been copied to the backup drive. The ~/Library/Preferences folder contained about 1/10 of the files it was supposed to contain, and numerous random files from all over the drive were missing, both in and out of the home folder. What was more disturbing was that, even after reformatting the TM drive, it started showing this same behavior little more than a month later. Things that have, at various points, gone missing from the backup:

• ~/Library/Preferences is by far the most common victim of this. If only one folder on the HD is suffering from being incompletely backed up, it's usually this one. Usually the affected files are for apps whose preferences have not changed in a long time.
• /Applications seems to get this a lot; sometimes applications are omitted outright (at one point, I was missing all the applications in /Applications that came after a certain letter in alphabetical order, 'O' if I remember correctly), sometimes files from inside their app bundles are missing, causing the apps to fail in mysterious ways.
• /System/Library/Framework seems to get hit by this a lot. This one isn't a big deal, since one could just reinstall the OS to get those files back, but it's still odd.
• ~/Library/Mail seems to get hit by this a lot, which is disturbing since for me, and probably for others as well, my mail archive is rather important for business reasons.
• Generally, folders that have files that have been sitting around for a long time without changing seem to be susceptible to this.

When the issue occurs, starting a new Time Machine backup doesn't always fix it. Sometimes, I've seen files continue to be missing even after forcing TM to do a complete file system traversal. About the only thing that's worked 100% of the time is to take a folder which is being incompletely backed up (in my case, ~/Library/Mail), rename it, do a backup, then rename it back, and do another backup. This causes TM to see the whole folder as new, and backs it up completely.

I've seen this issue on three separate machines now, at least for the case of ~/Library/Preferences. I'm wondering how widespread it is, so I was wondering if anyone here could try the following:

1. Run a manual Time Machine backup to bring your backup up to date.

2. Navigate to Library/Preferences inside your home folder. The Library folder inside your home is hidden by default, but you can get to it by holding down the Option key while choosing "Library" from the Go menu in the Finder.

3. If the status bar is not currently showing, choose "Show Status Bar" from the View menu to show it. Note the number of files in the Preferences folder.

4. Enter Time Machine.

5. Navigate to your most recent backup. Check the number of files in the version of home/Library/Preferences in your latest Time Machine backup.

6. If the number of files in the backed-up version of home/Library/Preferences is much less than the actual number of files in home/Library/Preferences, stop. Otherwise, click the "up" arrow repeatedly, checking previous backups to see if the file count reduces sharply in any of them.

At the time of this posting, the current backup has the correct number of files in the Preferences folder, 2253. However, if I go back to yesterday, there are only 342. Could you guys do this little test and let me know if you get the same results?

I also made a small program that can scan one's hard disk and compare the contents against the TM backup, to see what files, if any, are missing.

http://www.charlessoft.com/tmcheck.zip

[Spheric Harlot]

Great info, with a useful tool, to boot.

You're great, sir.

I'll get right on it.

[Spheric Harlot]

Had a whole list typed up, but the server had expired my token, and Safari ate everything when I swiped back.

Everything looks fine here.
just a bunch of temp files, Caches, and Logs, as well as some other stuff in /private/var/, and the obvious — trashes, MobileBackups, fsevents, Volume mount points, saved application states, etc.

[Spheric Harlot]

Same under the other user. iPhoto thumbnails and Final Cut Pro Backups.localized are not being saved, nor are a bunch of Containers including some archived iChats, and there's a handful of Mail attachments that aren't for some reason, but beyond that, everything looks fine.

No issues whatsoever regarding ~/Library/Preferences/

[CharlesS]

Originally Posted by Spheric Harlot 

Had a whole list typed up, but the server had expired my token, and Safari ate everything when I swiped back.

Sorry to hear that. I hate when that happens.

Everything looks fine here.
just a bunch of temp files, Caches, and Logs, as well as some other stuff in /private/var/, and the obvious — trashes, MobileBackups, fsevents, Volume mount points, saved application states, etc.

Yeah, that stuff is pretty much expected to be not backed up.

Originally Posted by Spheric Harlot 

nor are a bunch of Containers including some archived iChats, and there's a handful of Mail attachments that aren't for some reason

Hmm, these things sound somewhat fishy for them to be missing, though. Interesting.

No issues whatsoever regarding ~/Library/Preferences/

That's odd, ~/Library/Preferences is the one that never seems to back up correctly for me. Currently, that folder on my disk has 2256 items in it, while my latest TM backup has 358 files there.

[akent35]

Sorry to hear about your issues, CharlesS. It's especially disappointing that it is due to problems with Time Machine. Do you have any idea how long this has been going on?

I have never used Time Machine, but instead depend upon SuperDuper! for my complete (and bootable) backups. Those backups have always, always been reliable and accurate. If you are absolutely certain that your current system on your old MacBook Pro is good to go, use SuperDuper! to make a complete backup. Then, do a restore of this backup to your new MacBook Pro. That should give you the system you want.

What you could do then is to use Time Machine on the new MBP to make a backup, and see if you are still having the same issues. If so, then there is something definitely wrong with Time Machine, and if it was me, I would no longer rely on TM for my backups.

[CharlesS]

The problem persisted even after I migrated the old MBP to the new MBP, then reformatted the TM disk and started a fresh backup from the new MBP. So the problem is either something intrinsic to Time Machine, or there's something about the files on my disk that is confusing it. If it's the latter, then I dunno, maybe I can figure out what's doing it eventually. If it's the former, then I'd like to collect some data to see how many others are having the same problem, and if enough people are, I'm going to file a Radar report.

If it comes to it, I can certainly move to another backup system, but it won't be SuperDuper. SuperDuper is a clone, not a backup, and completely unsuitable for what I'd want to do with it. This could be a good excuse to play with ZFS, though. It looks like ZEVO hasn't been updated in a long time and isn't compatible with Mavericks, sadly, but OpenZFS on OS X looks pretty up-to-date and recent. I'd like to do some good for the community by figuring out what's going on here and reporting it to Apple, though.

It's really a shame that ZFS didn't become the OS X file system way back in Leopard like was rumored. Time Machine would be awesome on ZFS.

[Doc HM]

It's wholly empirical evidence but when I use customers TM backups to migrate their macs rather than connect Mac to Mac I get way more issues Have had iPhoto libraries not copy over (or back up). Several times I've had the migrated user left with no permissions to view their own files or folders.

Target mode is more reliable but I have found MA under mavericks to be much less reliable than previous incarnations. Old Mac to new Mac migration used to be one if he jewels in Mac OS's crown. Now it's just a bit ropey.

[reader50]

It is possible when you restored from TM backup, it "restored" some system files. From an earlier system version. The description suggests the backupd process records the most recent changes, then erratically screws up linking in the unchanged files. Should backupd have (say) a framework replaced from an earlier OS version, it might choke on a minor HFS+ tweak introduced by Apple.

That's all I can think of. Other than a new TM bug, of course. If the above speculation is correct, an OS reinstall would fix it. And the system log might record the screwups.

[CharlesS]

Originally Posted by reader50 

It is possible when you restored from TM backup, it "restored" some system files. From an earlier system version. The description suggests the backupd process records the most recent changes, then erratically screws up linking in the unchanged files. Should backupd have (say) a framework replaced from an earlier OS version, it might choke on a minor HFS+ tweak introduced by Apple.

That's all I can think of. Other than a new TM bug, of course. If the above speculation is correct, an OS reinstall would fix it. And the system log might record the screwups.

The issue occurred prior to the migration. That's how I initially noticed it; files were missing on the new MBP after the migration, and when I looked on the backup drive to see what was going on, I noticed the files were missing there too. So the old MBP, which is almost six years old and which has seen Leopard, Snow Leopard, Lion, Mountain Lion, and Mavericks all installed on it since it was migrated, was exhibiting the problem. I'm currently in the dark as to why.

If you check out your ~/Library/Preferences folder in Time Machine, does it show the correct number of files?

[reader50]

I am on 10.5, since 10.6+ was never released for PPC. It seems doubtful the bug goes back that far.

Still ... 365 items in current prefs folder. 365 items in latest TM backup prefs folder. Sorry.