Welcome to the MacNN Forums.

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

You are here: MacNN Forums > Software - Troubleshooting and Discussion > Developer Center > wildcard file name matching

wildcard file name matching
Thread Tools
reader50
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 22, 2018, 03:37 PM
 
I'm trying to rm (delete) some files in a folder based on the first 2 characters, but only where the filename is 6 characters long. There are no filename extensions, just bare names. I've run out of terminal-fu trying to solve it.

example file names to delete:
1f**** or 20****

1f3 (skip)
1f3210 (delete)
234 (skip)
234567 (delete)

What the heck is this about?

I have a 25TB network TM backup volume, hosted by another Mac. As it's a network TM, it uses a sparsebundle image file. The image has grown past 16 TB. Using the default band size of 8MB, the /bands/ folder within the bundle now contains over 2 million files.

Near as I can tell, once it passed 2 million files in the folder, the hdiutil commands became unreliable. In particular, hdiutil compact stopped working. It worked when the allocated image size was smaller than 16 TB.

So even though the image contains only 12.3 TB, the image occupies its allocated size of 19.5 TB. And TM reports 5.5 TB free (free space on the host volume). Should I attach a volume larger than that for backup at some point, TM will try to make space. Perhaps by wiping backups. As TM does not shrink the allocated space during such wipes (that's how I got to 12.5TB used within a 19.5TB image), TM may wipe everything but the latest backup. Then stop, saying it cannot clear enough space.

I used to periodically shrink the allocated sparsebundle using "hdiutil compact", but that stopped working when I passed 16TB allocated.

I've repaired both the host volume and the mounted image - nothing wrong with either. Disk Utility refuses to do image conversions on the image. hdiutil can increase the virtual size, but not decrease it. "hdiutil convert (sourceimage) -tgtimagekey sparse-band-size=65536" might work to reduce the band file count below 300,000. But "convert" requires an outfile, and there is not enough free space for a 2nd image. Not unless I wipe most of my backups first.

I've used a defrag utility to compact the image content into the beginning of the image. So the upper part of the image is empty, primed for sparsing. I also tried shrinking the partition size to 12.4 TB, but still could not release the extra allocated bands. hdiutil (and TM) were happy to increase the virtual size back to the 25TB host size. But nothing releases those empty bands, so available space never goes up.

If I manually delete the band files from #2-million up, "compact" should work again. The files are named sequentially in hex, with #2 million being approximately "200000", though in fact everything from "17xxxx" and up is empty.

Finder can open the folder, but uses a human-readable sort on the Name. So the files are shown in a mixed order, unaware of hex naming:

1
1a
1ca
2
2a
2aa
12
12b

With hundreds of thousands of files to delete out of 2M+, this makes proper selection impossible. Simple terminal wildcards using the "rm" command do not work, because a wildcard "*" can substitute for multiple characters. And the lack of filename extensions means I cannot use the "." as an anchor in the expression, which doesn't help either.

OmniDiskSweeper has been running over 8 hours, trying to process the bands folder. Maybe it will offer a more programmer-friendly sort, though it's a long shot. Finder search windows cannot select the Name with enough specificity to start with "2", while requiring a 6-chr length.

I may have to wipe the volume and start backups all over again with a fresh image. And manually convert the image to larger slices, while it's still small. So I do not end up in the same fix when it passes 16TB next time.

But I'd still like a shot at fixing it in place. Perhaps with a regex variation of the rm command. Unfortunately, I haven't learned regex yet.
     
reader50  (op)
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 24, 2018, 03:13 PM
 
Update for anyone interested. No solution turned up, so I did it the hard way. I wiped my media backups again, so the actual content of the backup image would be smaller than the free space on the host volume. Smaller than 5.5 TB in my case. Then tried to convert to a new image file with bigger bands.

hdiutil convert WILL NOT WORK on the old image. It took over an hour of failed attempts to figure that out. Same problem as compact, I assume: too many band files in the folder.

I had to manually create a new image with 64MB bands. Both the create stage, and previous convert efforts involved a lot of trial and error. hdiutil is finicky about syntax, and the man page examples have errors. Probably from not being updated in a long time.

Here is what worked:

1) hdiutil create -size 20t -type SPARSEBUNDLE -fs HFS+ -tgtimagekey sparse-band-size=131072 image.sparsebundle

notes:

{a} size 20t (20TB) is actually a MAX size parameter. The bundle will not take up that much space when created. The first time TM runs, it will adjust your max size up to match the host volume size. So just specify it big enough for the data you are cloning today. I could have used "3t".

{b} type SPARSEBUNDLE has to be in ALL CAPS.

{c} fs HFS+ you have to specify a filesystem, which will also cause partitioning. If you don't, you cannot mount & partition the new image later. Disk Utility should offer to format the blank "drive" when you mount a blank image. Instead, DU will complain there are no filesystems present. And will only offer to eject, not to fix things.

{d} sparse-band-size is how many 512-byte blocks per band. 131072 = 64 MB. The default for new images has been 8 MB since OSX Leopard.

2) Mount the image. Use Disk Utility to reformat it to case-sensitive HFS+, which will also silently add the journal. The "fs" options offered in hdiutil could have gotten closer, except they do not work. man page out of date again.

3) Also mount the old image. Use a TM-safe clone utility to copy the old TM volume to the new one. I used SuperDuper. Took about 30 hours.

4) Eject the old image volume. Rename the new image volume to Time Machine Backups. Then eject it too.

5) Copy the name of the old image file, then delete the old file. This will take a long time, as it's actually deleting over 2 million band files. Finally, you'll recover free space on the host volume.

6) Rename the new image file to match the old one.

7) On your main machine, mount the host volume over the network. Then open a Terminal window.

8) sudo tmutil inheritbackup (<- note the trailing space. do not press Return yet)

9) drag the new image file into Terminal. That will enter its path.

10) press Return. Terminal will ask for your admin password.

11) Done, unless you have multiple systems using that network backup. If other Macs (or bootable partitions) use that backup, repeat steps 7-10 on each system.

Trigger backups as desired. It will take longer than usual the first time, as TM will do a full scan of each volume you back up. In my case it will take days to rebackup all the files I had to remove from backup to make room.

I probably should have selected a still-bigger band size. Even at 64MB, the bands folder will eventually grow to ~300,000 bands. Next time I do a TM migration, I'll bump the band size again.
     
andi*pandi
Moderator
Join Date: Jun 2000
Location: inside 128, north of 90
Status: Offline
Reply With Quote
Sep 25, 2018, 11:35 AM
 
I read this thread and my eyes crossed and my brain cried out in pain.
     
P
Moderator
Join Date: Apr 2000
Location: Gothenburg, Sweden
Status: Offline
Reply With Quote
Sep 26, 2018, 04:51 PM
 
I am sorry I missed this thread (airplane troubles, yadda yadda) because this one I know: if you use ? Instead of *, it will match exactly one character, so

rm 1f????

would do it.
The new Mac Pro has up to 30 MB of cache inside the processor itself. That's more than the HD in my first Mac. Somehow I'm still running out of space.
     
reader50  (op)
Administrator
Join Date: Jun 2000
Location: California
Status: Offline
Reply With Quote
Sep 26, 2018, 06:25 PM
 
I'll have to remember that trick. Thank you.
     
   
Thread Tools
 
Forum Links
Forum Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Top
Privacy Policy
All times are GMT -4. The time now is 07:01 PM.
All contents of these forums © 1995-2017 MacNN. All rights reserved.
Branding + Design: www.gesamtbild.com
vBulletin v.3.8.8 © 2000-2017, Jelsoft Enterprises Ltd.,