Welcome to the MacNN Forums.

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

You are here: MacNN Forums > Software - Troubleshooting and Discussion > macOS > Time Machine backs up unchanged files

Time Machine backs up unchanged files
Thread Tools
reader50
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 5, 2018, 04:41 AM
 
Say I have an 8GB movie file. TM backs it up the first time. Whenever I watch it, TM backs it up again during the next backup. If I skim through several episodes of a favorite TV series, the next TM run can be 50 GB.

Get-Info shows the modification date is unchanged. Only the Last-Opened date has changed. I've been seeing this happen for the last few years, across Yosemite, Sierra, HS - all using HFS+. I suspect it happens on all files. It's just noticeable after you've opened big files.

It burns space on my backup volume. But since I switched from local backup to a network drive, it got far more annoying. Now it's also keeping my network busy for extended times.

TM is supposed to only back up changed files. In practice, it backs up changed + opened (but unchanged) files. It happens if the opened files are on the boot drive, or on a data drive. Googling doesn't find anything on the subject.

You can test this by forcing a manual backup. Then open a few video files for a few seconds each. Then force another manual backup. If you click on the TM menu icon during a backup, it will show how much is being backed up. This info also appears in the TM pref pane during a backup. TimeTracker from CharlesSoft shows the evidence after the fact.

Has anyone else seen this happening?
     
P
Moderator
Join Date: Apr 2000
Location: Gothenburg, Sweden
Status: Offline
Reply With Quote
Sep 5, 2018, 09:44 AM
 
I actually disabled the "last opened" thing a long time ago (because I got an SSD back when reliability was terrible), and it has probably carried over into my new setup. I'll check when I get home.

If it is a specific file, you should be able to work around it by marking it read only, I think?

EDIT: Are you still on HFS+ or have you updated to APFS? Because I think HFS+ wrote the access time stuff in the directory file, but maybe APFS changed that. I know that HFS+ writing everything in the same directory file was an issue for threading, so it makes sens that they may have changed it.
The new Mac Pro has up to 30 MB of cache inside the processor itself. That's more than the HD in my first Mac. Somehow I'm still running out of space.
     
reader50  (op)
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 5, 2018, 12:33 PM
 
I'm still on HFS+ because I'm still all-HDs. And it's my media library, not a specific file. Every time I watch (usually rewatch) something in the evening, TM gets a workout after.

How did you stop the bad behavior?
     
P
Moderator
Join Date: Apr 2000
Location: Gothenburg, Sweden
Status: Offline
Reply With Quote
Sep 6, 2018, 09:19 AM
 
What I did was disable the access time recording. This is done with the mount command, specifically:

mount -vuwo noatime /

/ is for the root volume, change to the mount point for your other drive if it is somewhere else. -v is "verbose" so we get all the error codes, if there are any. -u is for remounting an existing volume. -w is to make sure it is writable, and -o noatime is the magic bit that disable logging the access time. This will only last until the next reboot. To make it stick, you either have to modify the /etc/fstab or install an autostarting object. I have seen reports that the autostarting object is a little finicky in more recent versions of the OS, but here is a description of how to make one:

https://dpron.com/os-x-noatime-multiple-ssds/

Note that I don't have your problem, and I don't know that this is the fix. It was just the first thing that came to mind.

Also: Can't you just make all the video files, and the folder they're in, read-only?
The new Mac Pro has up to 30 MB of cache inside the processor itself. That's more than the HD in my first Mac. Somehow I'm still running out of space.
     
reader50  (op)
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 6, 2018, 12:18 PM
 
I've ripped my entire video library to disk. All DVDs and BRs. Then organized by type, series, season, etc. My media volume has 792 folders today, which will continue to increase. When something I like is available for dirt cheap (used typically), I pick it up. It turns out used digital media is exactly as good as new digital media.

I'd rather not solve this folder-by-folder. I have used 'mount' before, and fstab, to prevent some partitions from mounting. Thanks for a better answer, I'll report back with the results.
     
reader50  (op)
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 6, 2018, 03:55 PM
 
Not looking good. The media drive is an Apple software RAID, which may be complicating things.

"mount -vuwo noatime /Volumes/Media" appears to work. mount shows noatime has been applied. But the Finder continues to show (and update) last-opened, and TM continues to back up unchanged files after reading. Apparently it's using a different extended attribute.

"xattr" on a file gives "com.apple.lastuseddate#PS", and I don't have a volume command to ignore "lastuseddate".

Attempt to mount volume read-only

"mount -vur /Volumes/Media" always fails with a "Resource busy" error. Even right after boot.
"mount -vuo rdonly /Volumes Media" - same result.
"mount -vurf /Volumes/Media" fails with the error "mount_hfs: -o force: option not supported".

Editing fstab to add the line "UUID={snip} /Volumes/Media hfs ro" causes the volume to not mount on boot.

It appears macOS does not want a RAID volume to be mounted read-only. I'm not seeing any other obvious interpretation.

Set files read-only

Tried experimenting on a folder. Set it read-only for everyone, then applied to enclosed items. Subsequent viewings of the files within did not trigger TM backups. But TM initially re-backs-up the folder due to the changes. Interestingly, Finder Info still gets updated for last-opened. But no TM rebackup.

Lock files

I tried locking just a folder, then opening a file contained in a subfolder. Just locking the few top-level folders would be manageable. But - TM still re-backs-up the file after viewing. Every file needs to be locked (locking folders is optional). There doesn't seem to be a GUI utility for recursive locking, so more Terminal-fu.

"chflags -R uchg /path/to/test/folder" works. All contents locked, and subsequent viewings do not trigger TM backups. But TM initially re-backs-up everything due to the change.

Conclusion

Apparently TM considers the extended attributes part of a file. So an attribute change triggers a fresh backup. I can solve it file-by-file with either read-only permissions or locking. Then ignore permissions on the volume when I make changes. I'll go the permissions route if I have to, to avoid all the lock icons.

But doing it via file changes will trigger a full TM rebackup of the media volume. It's over 9TB, so it will take a day or three over the network. I'll have to start by deleting my media backups from TM, as there isn't enough free space for another full backup. TM would erase a year of my oldest backups to clear enough space.

I'd really like a read-only volume mount solution before diving into days of backups. So I'm going to hold off a few days, in case someone thinks of a better answer.
( Last edited by reader50; Sep 7, 2018 at 12:59 AM. Reason: clarify)
     
P
Moderator
Join Date: Apr 2000
Location: Gothenburg, Sweden
Status: Offline
Reply With Quote
Sep 7, 2018, 07:07 AM
 
What happens if you unmount the drive first, and then mount it manually as read only? The unmount command might fail if there are open files, but if so, you can find the open files with "lsof | grep".

Or you just exclude the entire media library from Time Machine and make a manual backup. If it isn't going to change anyway, it might be the easiest fix.
The new Mac Pro has up to 30 MB of cache inside the processor itself. That's more than the HD in my first Mac. Somehow I'm still running out of space.
     
reader50  (op)
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 7, 2018, 12:53 PM
 
Manual-unmount worked. But when I told Disk Utility to remount, nothing happened. It took a reboot to remount the volume. I didn't investigate enough to establish why. By that point I'd seen the pattern - there's an unexpected failure every way I try to mount read-only. With multiple methods.

The grep command returns no open files. The resource is busy before and after issuing the grep command.

I do want the media volume backed up, and it does change. Not daily usually, more like monthly. I could exclude it most of the time, then let TM catch up once a month. But all files opened in the interim would then back up. It could make for a 1TB backup pass. And I'd have to manually manage TM. Backups are reliable when they are fully automatic.

The better answer is to stop TM from seeing opened files as needing backup. I'm going to apply the permissions changes using BatChmod and pay the price once with a full backup.
     
reader50  (op)
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 9, 2018, 09:17 PM
 
Media drive backup completed with restricted permissions.

I did some brief testing with an APFS volume. Same result - opening a file triggers a fresh TM backup on the next pass. Switching the mount to noatime did not change anything - more backups of unchanged files.

TM must be doing backups based on changes to the extended attributes. I've solved my problem, in a somewhat inconvenient way. But a better solution is needed. The average person shouldn't have to get into the nuts and bolts of *nix permissions to solve a nuisance problem.

Edit - forgot to mention. I used BatChmod with full recursion to set every file and folder on my media volume to root:wheel, with Write permission denied to group and everyone. Then did another recursive change, but applied to folders only: 777 (all permissions allowed).

This way I can add new files to folders, or edit folder names without authenticating. But I do have to update owner/group/permissions whenever I add a file, or that file will backup whenever I watch it. Authentication is needed if I change Finder flags on a file, etc. This combination stops the nuisance rebackups, with reduced authentications.
( Last edited by reader50; Sep 9, 2018 at 11:07 PM. )
     
reader50  (op)
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 9, 2018, 09:27 PM
 
While snooping around my problem, I found an interesting note in the tmutil man page:
verifychecksums path ...
Compute a checksum of data contained within a backup and verify the result(s) against checksum information computed at the time of backup.

No output is generated for matching checksums. Issues are reported using the following legend:

! The file's current checksum does not match the expected recorded checksum.
? The file's recorded checksum is invalid.

Beginning in OS X 10.11, Time Machine records checksums of files copied into snapshots. Checksums are not retroactively computed for files that were copied by earlier releases of OS X.
There are no other checksum mentions within the tmutil man page. No command to manually add them to older snapshots, for example.

So macOS does compute and save file checksums, though only within a TM backup. It might be interesting to have a utility that could fill in the checksums for older files. Then do a file-level dedupe of matching files. Even under HFS+, a file dedupe would be safe within a TM backup.

Such a utility would only solve part of my problem. It wouldn't address the nuisance backups and network activity.

But it could retrieve a lot of wasted space on any TM volume. Even if the extended attributes are included in TM's checksums, you still get back space from files you've moved around. Whenever I organize folder trees, TM does its thing, backing up unchanged files.
     
   
Thread Tools
 
Forum Links
Forum Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Top
Privacy Policy
All times are GMT -4. The time now is 04:20 PM.
All contents of these forums © 1995-2017 MacNN. All rights reserved.
Branding + Design: www.gesamtbild.com
vBulletin v.3.8.8 © 2000-2017, Jelsoft Enterprises Ltd.,