[FireWire]

I was surfing the 'net earlier this evening, when I noticed my external HD being used for an unusual amount of time. It is plugged via FireWire 400 and is used exclusively for Time Machine backups. I opened the Time Machine preference pane and I noticed that it was backuping over 5 GB worth of data!

I'm 100% positive that I didn't download or install anything in the hours prior to this backup: I was merely using my computer (reading articles on Wikipedia, chatting via MSN). I can't see where these 5 GB come from. Is there a way to know what was backuped recently by TM?

[Hal Itosis]

Originally Posted by FireWire 

I was surfing the 'net earlier this evening, when I noticed my external HD being used for an unusual amount of time. It is plugged via FireWire 400 and is used exclusively for Time Machine backups. I opened the Time Machine preference pane and I noticed that it was backuping over 5 GB worth of data!

I'm 100% positive that I didn't download or install anything in the hours prior to this backup: I was merely using my computer (reading articles on Wikipedia, chatting via MSN). I can't see where these 5 GB come from. Is there a way to know what was backuped recently by TM?

The 'Spotlight' section in this article shows a way to do it with mdfind... but
we need to calculate a really oddball time offset value (a script is provided).

[sigh... when it comes to Spotlight/mdfind, why doesn't Apple allow regex, and more familiar UNIX-like 'find' syntax?]

Anyway, the article *briefly* touches on how sometimes Time Machine can get 'out of sync' and
mentions messages like "node requires deep traversal" reason:kFSEDBEventFlagMustScanSubDirs|
kFSEDBEventFlagReasonEventDBUntrustable| and other stuff I can't quite grasp yet. Good luck.

--

But really... even a "time offset" isn't necessarily the right answer. It would be nice if Time Machine provided
a simple way to determine which files were processed [i.e., copied and/or re-linked and/or unlinked?] on its
last run. Am I missing something? Perhaps inode access times, or some extended attribute? idunno. Anyone?

[Hal Itosis]

Originally Posted by FireWire 

I'm 100% positive that I didn't download or install anything in the hours prior to this backup

Also note that large transfers don't necessarily have to represent 'new' (or even 'modified') items. The article gave an example of renaming a file... and how that gets treated as being a different file, than its previous incarnations [despite having identical content and inode number].

Now imagine a folder... say (for example) named "my docs from the 1990's" with a few gigs worth of ancient content. What happens if we rename that folder to "my 90s docs"? Does Time Machine simply *rename* the corresponding folder in its backup database? Or... does it recopy all that (old, unchanged!) data into what it perceives as being a new directory/hierarchy? [and a similar treatment would likely occur when handling moved items.]

[CharlesS]

This intrigued me, so I made a little app to answer FireWire's question.

http://www.charlessoft.com/TimeTracker.zip

Click on any Time Machine backup and it should compare that backup with the previous one to see what changed. It does not distinguish between freshly backed-up files and moved files, though, so it won't answer Hal's question. However, going in with the Terminal and using ls -i to check the inode numbers of some moved files does show that the inodes are different, indicating that the files were re-backed up rather than simply hard-linked.

[turtle777]

Originally Posted by Hal Itosis 

Now imagine a folder... say (for example) named "my docs from the 1990's" with a few gigs worth of ancient content. What happens if we rename that folder to "my 90s docs"? Does Time Machine simply *rename* the corresponding folder in its backup database? Or... does it recopy all that (old, unchanged!) data into what it perceives as being a new directory/hierarchy? [and a similar treatment would likely occur when handling moved items.]

That's indeed an interesting question.

From experience, the TM volume gets bloated pretty fast.
I can not but help that TM is NOT very smart about renamed or moved files, and that this causes the bloating.

-t

[Simon]

I agree. My experience with observing the remaining available space on the TM volume is if you rename or move something TM will back it up again. It appears TM's not (yet) smart enough to recognize renaming or moving.

[Hal Itosis]

Originally Posted by Simon 

I agree. My experience with observing the remaining available space on the TM volume is if you rename or move something TM will back it up again. It appears TM's not (yet) smart enough to recognize renaming or moving.

I think most backup utilities do [mis]handle renamed items in a similar fashion.

The only one I can recall that made a point of pre-processing/detecting renaming was PowerMerge (which was the main competitor to Dantz's DiskFit at the time... iirc). There is an inherent amount of overhead in pulling this off I guess, because it needs to store a list of names along with inode (or catalog id?) numbers, to accurately assess the situation. Unfortunately, PowerMerge never made the transition to OSX.

[Hal Itosis]

Originally Posted by CharlesS 

This intrigued me, so I made a little app to answer FireWire's question.

http://www.charlessoft.com/TimeTracker.zip

Click on any Time Machine backup and it should compare that backup with the previous one to see what changed. It does not distinguish between freshly backed-up files and moved files, though, so it won't answer Hal's question. However, going in with the Terminal and using ls -i to check the inode numbers of some moved files does show that the inodes are different, indicating that the files were re-backed up rather than simply hard-linked.

NICE ONE! Thanks... this will be useful and informative.

[JellyBeen]

Thanks Charles for whipping-up this little app!

[MacosNerd]

Originally Posted by CharlesS 

This intrigued me, so I made a little app to answer FireWire's question.

http://www.charlessoft.com/TimeTracker.zip

Great little app there Charles.

[turtle777]

Originally Posted by CharlesS 

This intrigued me, so I made a little app to answer FireWire's question.

http://www.charlessoft.com/TimeTracker.zip

Awesome

It helped me identify a huge waste of HD space.

Thanks a lot

-t

[Simon]

Great little app indeed.

[idea2go]

Originally Posted by CharlesS 

This intrigued me, so I made a little app to answer FireWire's question.

http://www.charlessoft.com/TimeTracker.zip

Click on any Time Machine backup and it should compare that backup with the previous one to see what changed. [...].

All I get is a blank 3 column window (nothing to click on). Could this be because I use Time Capsule?

[slpdLoad]

Originally Posted by Hal Itosis 

Now imagine a folder... say (for example) named "my docs from the 1990's" with a few gigs worth of ancient content. What happens if we rename that folder to "my 90s docs"? Does Time Machine simply *rename* the corresponding folder in its backup database? Or... does it recopy all that (old, unchanged!) data into what it perceives as being a new directory/hierarchy? [and a similar treatment would likely occur when handling moved items.]

It would appear to be doing that, but from what I've read it's actually not making more physical copies on the platter. It uses something called hard and soft links to refer to data from multiple places and never delete it until the last link is removed. Here's some interesting reading (the whole article is great):

AppleInsider | Road to Mac OS X Leopard: Time Machine [Page 3]

[Hal Itosis]

Originally Posted by slpdLoad 

It would appear to be doing that, but from what I've read it's actually not making more physical copies on the platter. It uses something called hard and soft links to refer to data from multiple places and never delete it until the last link is removed.

 
'fraid not. The linkage advantage doesn't apply to moved items. Only unaltered, unmoved items get hard-linked. I just ran a test which demonstrates this. On my desktop since last night was a clipped screenshot of some music files I intend to download later. Named "cart.png"... it has been there during several backup runs. At 2:30PM today (after reading your post) I moved it off the desktop into my home folder, and waited for the 3PM backup run... after which, I ran this command:


ls -lion /Volumes/LaCie/Backups.backupdb/HAL/2008-04-24-{0902*,1402*,1502*}/MacX/Users/halito/{,Desktop/}cart.png

ls: /Volumes/LaCie/Backups.backupdb/HAL/2008-04-24-0902*/MacX/Users/halito/cart.png: No such file or directory
ls: /Volumes/LaCie/Backups.backupdb/HAL/2008-04-24-1402*/MacX/Users/halito/cart.png: No such file or directory
ls: /Volumes/LaCie/Backups.backupdb/HAL/2008-04-24-1502*/MacX/Users/halito/Desktop/cart.png: No such file or directory

3138689 -rw-r--r--@ 6 501 54968 Apr 23 23:01 / . . . /2008-04-24-090250/MacX/Users/halito/Desktop/cart.png
3138689 -rw-r--r--@ 6 501 54968 Apr 23 23:01 / . . . /2008-04-24-140233/MacX/Users/halito/Desktop/cart.png
3145654 -rw-r--r--@ 1 501 54968 Apr 23 23:01 / . . . /2008-04-24-150235/MacX/Users/halito/cart.png


After running that blue command, I got 3 (expected) errors. First 2 say the cart file isn't in the 9AM or 2PM home folder (it wasn't ). The 3rd error says the cart file isn't on the 3PM desktop. Also true. Now look at the inode numbers and the link counters. The 9AM and 2PM have identical inodes and a count of 6 (associated with 4 other hourly backups). The 3PM cart file has a different inode and only 1 link (itself).

Now Time Machine could have used a hard link for this... but it would require extra (time-consuming) file checking.
[i also think it might introduce other complexities, but I don't have a clear well-thought-out example of that yet.]

--

Shucks... need to test again for *renaming* [but i think it's the same.]

--

Yup. I simply renamed cart to kart and waited again. Here's the result (new inode, singly-linked):

ls -lion /Volumes/LaCie/Backups.backupdb/HAL/2008-04-24-{1502*,1602*}/MacX/Users/halito/[ck]art.png
3145654 -rw-r--r--@ 1 501 54968 Apr 23 23:01 / . . . /2008-04-24-150235/MacX/Users/halito/cart.png
3146353 -rw-r--r--@ 1 501 54968 Apr 23 23:01 / . . . /2008-04-24-160229/MacX/Users/halito/kart.png

[CharlesS]

Originally Posted by idea2go 

All I get is a blank 3 column window (nothing to click on). Could this be because I use Time Capsule?

Probably. I don't have a Time Capsule, so I wasn't able to look and see how they work, how to browse the files on one, where Time Machine's plist file stores the info designating where it is and what its address is, etc.

[Drakino]

Originally Posted by CharlesS 

Probably. I don't have a Time Capsule, so I wasn't able to look and see how they work, how to browse the files on one, where Time Machine's plist file stores the info designating where it is and what its address is, etc.

You can simulate having one pretty easily if you have two Macs handy. Turn on file sharing on one, then point the other machine to it in Time Machine. I don't have a Time Capsule either, but poking at what it was doing at the Apple store was revealing pretty much the same behavior as backing up to another AFP machine.

I tried running it here but got the same result, even when I mounted the Time Machine backup manually. If you want help testing it, I'm willing to try out new versions. It could be a pretty handy little utility.

If you want a quick fix, you can tell people to manually mount the backup (easily done by entering Time Machine then backing out, as it doesn't unmount for me till the next time a backup runs), and looking for the mount in /Volumes titles "Backup of <machine name>".

[CharlesS]

Having two Macs is not a problem. Having two Leopard-capable Macs available for testing purposes and an 802.11n router is, although that may change sometime in the future.

I'm curious - how exactly does Time Machine over AFP work? What you're saying is that you don't even have to have the disk mounted? Does it store your user/pass in the Keychain and just log into it when necessary? If you take your laptop out of range of the router, and then come back home later on, does it automatically just reconnect back to the router without your having to mount the AFP server again? If so, that's pretty nifty.

[steve626]

Originally Posted by Simon 

I agree. My experience with observing the remaining available space on the TM volume is if you rename or move something TM will back it up again. It appears TM's not (yet) smart enough to recognize renaming or moving.

I think you're right about that at least so far, Time Machine is not really sophisticated about these things. However other backup and cloning programs often do the same thing. If you think about it from the perspective of the person developing the backup software, doing a lot of such checks on every file and folder encountered will hurt performance overall and at some point it might be judged better just to make it simpler (fewer bugs to worry about introducing) and err on the side of checking less and copying more. It's a trade -- the more cleverly the program checks, the slower it runs in general and the more complex the code will be (and the greater the chance of complications or bugs introduced). This is to balanced against using more disk space etc.

[Jasoco]

Time Machine is indeed stupid. If you rename the folder or file, it does indeed recopy them. Creating in essence two identical copies of the file or folder you renamed. This creates a huge waste of space if you aren't thinking.

I hope TM will evolve. I mean the OS can track a file if it's renamed or moved. Why can't the Time Machine do the same?

[Drakino]

Originally Posted by CharlesS 

Having two Macs is not a problem. Having two Leopard-capable Macs available for testing purposes and an 802.11n router is, although that may change sometime in the future.

I'm curious - how exactly does Time Machine over AFP work? What you're saying is that you don't even have to have the disk mounted? Does it store your user/pass in the Keychain and just log into it when necessary? If you take your laptop out of range of the router, and then come back home later on, does it automatically just reconnect back to the router without your having to mount the AFP server again? If so, that's pretty nifty.

Basically Time Machine over a network (to a Time Capsule, another Leopard machine, or via the unsupported flag to any SMB/AFP server) will automatically mount and unmount both the share and the sparseimage it creates when it is needed. So if my laptop is at home either wired or wireless, it can see my AFP serving NAS, and work as normal. The share never shows up in the finder as mounted, but the sparseimage will show up. The username and password ends up being stored as a system level keychain item so it can work for any user logged into the system.

This is what I see if I type mount in the terminal when a backup is occurring

afp_35BeRG35BeRG35BeRG35BeRG-2.2f000018 on /Volumes/TM_MBPro (afpfs)
/dev/disk2s2 on /Volumes/Backup of abraxas (hfs, local, nodev, nosuid, journaled)

I've even enabled the AFP port to pass through my router to my NAS, and when I manually mounted the Time Machine share via afp://internet.ip , Time Machine even worked to backup over the internet, albeit very slowly.

[CharlesS]

It creates a sparseimage, then? Huh, that's odd. On my wired FW800 backup drive, it just puts everything in a regular folder on the root of the backup drive.

That is pretty neat that it mounts and dismounts as needed, though. I had always figured it would need to stay continually mounted to work.

[Drakino]

The sparseimage makes sense in that you no longer have to worry about the file server supporting hard links on directories. The file system and how to read/write to it becomes the ownership of the local machine, instead of the remote one. I was surprised the release version forced people to only use a Leopard system over the network, then eventually Time Capsule. I have it working fine to an AFP 3rd party NAS at home, and an SMB NAS at work. The downside to these solutions is that you have to mount them manually if booting off a Leopard DVD to do a full reinstall, but I'll deal with that minor inconvenience to put my backups where I want.

[Simon]

Originally Posted by CharlesS 

It creates a sparseimage, then? Huh, that's odd. On my wired FW800 backup drive, it just puts everything in a regular folder on the root of the backup drive.

Yep indeed. TM works different on TC than on regular 'local' volumes. The main difference is actually sparseimages. It's been reported by many people (on this board too).

I figure this is also why TT doesn't work with TC right now. If you launch TT w/o having mounted your TM backup partition you get the empty column window. It's the same for TC users. Since their backup volume isn't mounted (TC does that automatically when needed), TT doesn't see it and displays the empty window. For TC users TT would have to mount the TC volume first. And then of course work with the sparseimages.

[CharlesS]

I've put up a new build of TimeTracker that should (hopefully) work with AFP-connected drives and Time Capsules:

http://www.charlessoft.com/TimeTracker.zip

I haven't been able to test either of those things myself, since I only have a backup drive connected via FW800, but Drakino was kind enough to test this for me, and he says it's working. He also helped me troubleshoot a few AFP-related issues, so everyone who appreciates this build, give a high five to Drakino.

[Drakino]

Glad to help. It's a handy little utility for seeing exactly what Time Machine is doing, and has already been useful to my on my new system.

Thanks for adding in the remote time machine feature even without any direct way to test it on your side.

[JKT]

Folks will probably find this interesting then:

TidBITS Blog Post: Prune Your Time Machine Backups Selectively

N.B. Follow the link for the modified version of Grand Perspective - not the one to the official version mentioned in the "GrandPerspective and WhatSize Identify Disk Pigs" article.

(The modified version understands the way Time Machine links files in its database and can parse the information for you, the unmodified version does not and will churn away for a very long time until it eventually crashes).

The software provides you with a graphical view of what has been backed up so that you can spot e.g. large items that are using up space, or items that have multiple versions backed up which you don't really need to keep. You can then use Time Machine to remove them.

[CharlesS]

Hehe, that doesn't look as easy to use as my solution.

[JKT]

You're right, but it does make it easy to spot large to really huge files or packages that have been backed up multiple times and see what they are more easily. Your solution is good for comparing individual back-ups with each other in an OmniDiskSweeper/WhatSize fashion. However, the Grand Perspective solution is good for getting an overview of all back-ups at once (and especially so for the big files) and it is quicker for finding out if you really do need e.g. 20 backups of your Parallels virtual image or not.

[Jasoco]

Bump: CharlesS, our TimeTrackers have expired! Can we get an updated version? There isn't one listed on http://www.charlessoft.com/. I need my TimeTracker!

[CharlesS]

I already updated the file that's pointed to by the link in this thread, over two weeks ago.

[Jasoco]

I was just going by the URL in the app itself. It pointed me to a site with no link to the updated version.

Thanks.

[CharlesS]

Yeah, sorry about that. At the time, I expected to have made TimeTracker "official" by now, but some other things ended up preempting it on the priority list.

[Jasoco]

No problem. You know what else would be cool, an official icon instead of just the generic Application icon. Maybe something like... the Time Machine icon with a magnifying glass or something on it.

[CharlesS]

I'm sure that would be cool, but unfortunately I have absolutely no Photoshop skills, so I'd need someone to be generous enough to donate me an icon.

Since TimeTracker officially doesn't exist yet (it's intended just to be a sneak preview thing for MacNN users), it can probably get away with no icon for the short term.

[andreas_g4]

Charles, a bag of thanks to you for your nice little app, I much appreciate it.

[43skidoo]

CharlesS: nice app. Many thanks for that. It's proving to be useful to me already!

[Hal Itosis]

Originally Posted by CharlesS 

Since TimeTracker officially doesn't exist yet (it's intended just to be a sneak preview thing for MacNN users), it can probably get away with no icon for the short term.

Hmm, look what popped up on the radar yesterday (Friday): <Back-In-Time>
Seems to have a few features, but i'm not sure it displays the unique contents
of each snapshot like TimeTracker does. [which i still dig a whole lot... thanks.]

[CharlesS]

^ Warning: this app launches itself as root (and sets its setuid bit) when you launch it instead of using a helper tool, which is an extremely, extremely bad practice.

And AFAICT, it doesn't seem to do what TimeTracker does, which makes it not make sense in this thread. It seems to be competing more with Time Machine's built-in functionality, just in a really, really insecure way (and requiring you to manually turn Time Machine off before using it, to boot).

Thanks, I'll pass.

[Hal Itosis]

Thanks, I'll pass.

<aol>

me too

</aol>

[turtle777]

Back-In-Time doesn't do what TimeTracker does, but it looks like a very useful app.

However, $ 49 is quite steep, and that it launches in root is a problem.
The later might be fixed in an update, but the price is just too much.

-t

[schalliol]

TimeTracker is a great app! After missing finding this through a search I posted this thread which basically asks for TimeTracker functionality.

It would be cool if the app could place a file size next to the index file name to tell you how much new data was written during each backup so you know when you should look in more detail.

[CharlesS]

It's a nice idea, but to do that, I'd probably have to scan all the backups ahead of time instead of just doing it on the fly when you click on a backup, which would take a long time if you've got a lot of backups.

[Jasoco]

You could could do what the Finder does and fill in the blanks over time as they are calculated. Then save those sizes for later on. After the initial calculations the sizes would be remembered and it would only need to calculate newer ones.

[CharlesS]

I know, but it would cause a lot of disk churning that would go on for quite a while after you launched the app, and would probably slow things down when you clicked on a backup to view it.

Yeah, I know, I could pause the background process when you clicked on something, and I could cache data from earlier launches... I'm just being lazy right now. It might actually be a pretty cool idea if I have some free time later.

[Jasoco]

That's why it would be a low priority thing and could do the calculations in its own idle time from newest to oldest. (As the oldest would be the biggest most of the time.) The Finder does it this way with size calculation being low "in your own time" priority.

[schalliol]

Originally Posted by CharlesS 

It's a nice idea, but to do that, I'd probably have to scan all the backups ahead of time instead of just doing it on the fly when you click on a backup, which would take a long time if you've got a lot of backups.

When I click on a backup event, you show the new data on that backup. Could you just store that data next to the date of the backup? It would only be stored after the user clicks on a backup. As such, if someone wants a complete index, they would have to click on each backup at least once.

[CharlesS]

Originally Posted by Jasoco 

That's why it would be a low priority thing and could do the calculations in its own idle time from newest to oldest. (As the oldest would be the biggest most of the time.) The Finder does it this way with size calculation being low "in your own time" priority.

Yeah... actually, since this thing requires Leopard anyway, I could use all that newfangled NSOperationQueue stuff to do this instead of doing the threading by hand the old fashioned way. I'll have to have a look at this when I have the time.

[schalliol]

Cool, let me know if you'd like any testing if you get around to that!

[CharlesS]

Oops, looks like I won't be using NSOperationQueue after all