[VValdo]

Before the mods lock this forum for being a "tiger support question", let me establish that I don't have Tiger, I've never seen Tiger running, but I'd like to know more about some of the technology it will offer.

Here's my question.. I've read a bit about how Spotlight will work, and was wondering how it will relate to privacy & security. It seems that the indexing process requires copying portions of all searchable file's context into the index(es). When a file is indexed, where is the index located and what kind of permissions are necessary to view it? Is it encrypted in any way? Can the original file be reconstituted from the index? Can one user do searches on the files of another user, etc?

I read in another thread here that you will be able to flag certain folders so that they are not indexed. I assume this is true of FileVaulted directroy contents as well (hopefully not indexing is the default behavior). Is there a way to tell Spotlight to not index certain file types or certain attributes/metadata, etc?

I'm sure these issues have been given a lot of thought and have easy answers, but I hadn't heard much about this...

Thanks,
W

[Hi I'm Ben]

Spotlight is neat. You will like it.

here is the spotlight pdf on apple's site

[Agent69]

Actually, I am more interested if it can be turned off system wide. I personally have no need for Spotlight (although I think it will be useful for others).

[CubeWannaB]

Someone else suggested that spotlight can be disabled at the command line. I don't have Tiger so I can't verify this myself.

However, if it is true this may have unforeseen consequences. As time goes on an increasing number of applications are going to need Spotlight. You may argue that you don't want to search your email or graphics files, but when Tiger has been around for a while and Mac developers come to rely on Spotlight your programs may not even launch without it.

As a developer I can imagine lots of things to do with Spotlight that may not be immediately obvious. An installer could use it to locate older versions, something which currently takes the Adobe installer a long time. A dictionary could make each word a file and use Spotlight for looking up words.

These aren't the best examples - I came up with them on the spot - but I hope they show that Spotlight is going to have larger repercussions that just personal file searches. Turning it off isn't going to be feasible for very long.

[VValdo]

That's a really good point. If spotlight is ASSUMED to be on, what kind of ramifications will this have for applications that use it?

W

This is an offtopic question, but does anyone know if Tiger fixes the FileVault security holes? The first was passwords showing up in the swap files (well known), the slightly more obscure one was a weakly hashed password in /var/db/shadow (as described here). I hope these two were fixed as well.

[VValdo]

I also hope that FileVault's default action is to prohibit indexing in secured areas-- unless the index itself is encrypted as well. In which case I assume that other users/applications would never have access to the encrypted index. I would love to see a white paper type document on how spotlight works and respects permissions, encryption, etc.

W

[ajbaker]

There have been a good number of posts regards Spotlight and Security - I think most boil down to a number of questions which no one has explicitly asked and I myself have not seen answered.

Presumably Spotlight stories its index in one file per disk?

If this is correct, presumably the index stores data for files in all user folders, including FileVault protected user folders?

If this is true, if I perform a Search using Spotlight will it return (1) data from all users' data even if I do not have permission to view the file
OR (2) will it return data on only files I have read access to (even if they are in another users folder)
OR (3) will it onyl return data for files in my user folder?

[JLL]

Originally Posted by Agent69

Actually, I am more interested if it can be turned off system wide. I personally have no need for Spotlight (although I think it will be useful for others).

You never search for anything?

[Agent69]

Originally Posted by JLL

You never search for anything?

No, I have no need to. I keep my home folder organzied.

[SMacTech]

Originally Posted by Agent69

No, I have no need to. I keep my home folder organzied.

Drag your disks to the privacy tab and it won't be spotlighted !!

[jasong]

Originally Posted by ajbaker

Presumably Spotlight stories its index in one file per disk?

*Edit* this is correct, my bad

(1) data from all users' data even if I do not have permission to view the file
OR (2) will it return data on only files I have read access to (even if they are in another users folder) OR (3) will it onyl return data for files in my user folder?

None of the above, Spotlight isn't just about user folders, it can be used to find programs as well. Presumably (which means this could be wrong) spotlight runs as a user process, which means that it has access to whatever you have access to. It can't read other users folders just like you can't read them (unless you are root? - which you shouldn't be), but it can index the Applications folder, etc.

[JLL]

Originally Posted by Agent69

No, I have no need to. I keep my home folder organzied.

And mails, images, music, contacts?

[Silky Voice of The Gorn]

Spotlight is a low-level function of Tiger, I don't know why anyone would want to disable it. It would be like disabling Quartz because you don't watch movies in Quicktime player.

[Hi I'm Ben]

Originally Posted by Silky Voice of The Gorn

Spotlight is a low-level function of Tiger, I don't know why anyone would want to disable it. It would be like disabling Quartz because you don't watch movies in Quicktime player.

I think a part of people want to disable it to show off just how badass they are. As stated before "I don't need it I have my home folder organized" Yeah, me too buddy, I guess I just find it easier to get to 1 file out of 10,000 by typing one word rather then navigating to it, call me lazy, I'll just say you're ignorant to technology.

And my god could you guys freak out just a little more about an app that indexes. Bottom line is, if you duon't want it showing the index to certain areas, it won't. And YES it will index your entire drive and keep it in a file, are you worried your 14 year old kid is going become a mastercracker decode your index file and find out about your porn stash? Or someone from Russia is going to break your firewall and steal the index file from your computer containing all your financial locations of files.

Don't want your files indexed. Drag your stupid homefolder into the ignore section, drag your top level system files, tell spotlight not to index programs, do a plethora of paranoid things until anyword you type into spotlight turns up 0 results.

Since to most spotlight is a feature, I'm glad it's on be default. The overly paranoid and people who just like to complain will certainly figure out how to hide their oh so secret files.

[ajbaker]

Originally Posted by Hi I'm Ben

And my god could you guys freak out just a little more about an app that indexes. Bottom line is, if you duon't want it showing the index to certain areas, it won't. And YES it will index your entire drive and keep it in a file, are you worried your 14 year old kid is going become a mastercracker decode your index file and find out about your porn stash?

No need to be quite so critical. We have no experience of Tiger and are trying to understand it before it is released. Nothing wrong with that.

Neither is there anything wrong expecting Spotlight not to show the existence of another users private files. After all, thats what permissions are there for! I doubt the "Dont Index this directory" tab is designed to prevent indexing another users home, I guess it's designed for directories in YOUR home.

So back to the original question that your overtly critical post failed to answer (but others have been kind enough to add their thoughts): if Spotlight has one index per disk, does it also index file permissions so that it will not display anothers users files (if you do not have access).

[Hi I'm Ben]

Originally Posted by ajbaker

No need to be quite so critical. We have no experience of Tiger and are trying to understand it before it is released. Nothing wrong with that.

Neither is there anything wrong expecting Spotlight not to show the existence of another users private files. After all, thats what permissions are there for! I doubt the "Dont Index this directory" tab is designed to prevent indexing another users home, I guess it's designed for directories in YOUR home.

So back to the original question that your overtly critical post failed to answer (but others have been kind enough to add their thoughts): if Spotlight has one index per disk, does it also index file permissions so that it will not display anothers users files (if you do not have access).

No it will not show another users files. Spotlight is user aware, it only searches where it has premission to search.

[Millennium]

As I understand it, FileVault is likely to work as follows:

1) If you do not use FileVault, then your home folder will be put into the systemwide index. It may or may not be searchable by other users; anyone who has tested this would be probably violating an NDA if they were to confirm or deny it.
2) If you use FileVault, then Spotlight will try to put your Home "folder" (actually an encrypted disk image) into the systemwide index, but because it's encrypted Spotlight will see only junk. It cannot decrypt the file, because it doesn't have your password.
3) When you log in, your Home image will be indexed as though it were a disk (since, after all, it's a disk image). Depending on the permissions for this index file, it should be accessible only to you. Even if Apple screws this part up, it should be possible to fix it with little trouble. This is a case where ACLs could come in handy: allow full access to the owner and whatever user FileVault runs as, but no access for anyone else.

Bottom line: If you don't want your Home folder searchable by others, simply turn FileVault on and you're done. That will definitely stop other users from getting to your content.

[Silky Voice of The Gorn]

Spotlight is unobtrusive, fast, and "just works". The initial indexing after installing Tiger takes a few minutes, but it happens behind the scenes and doesn't slow you down. It does NOT index other users, nor can you search other users ~/, except for their public and sites folder. In other words, Spotlight can't see what you can't see.

Filtering out folders for indexing via the Spotlight Privacy tab is useful if you're worried about little Johnny searching those folders with your account logged in...but of course those same folders can be found manually through the Finder, so if Tinfoil is your headwear of choice, better go with Filevault or encrypted dmg's...

I find it interesting that people here are assuming that Apple did it wrong rather than right. What's up with that?

[Macrat]

I'm with Agent69, I have no use for Spotlight. I am more concerned about the performance hit with it on, I would rather have it off with no possible slowdown on my old powerbook, unless it is truly speedy enough not to matter.

[Millennium]

Originally Posted by Macrat

I'm with Agent69, I have no use for Spotlight. I am more concerned about the performance hit with it on, I would rather have it off with no possible slowdown on my old powerbook, unless it is truly speedy enough not to matter.

There should be no slowdown at all, except for the initial install. Assuming that Spotlight is properly implemented -and this appears to be the case- files should only be indexed when they are saved, rather than having to go through a background process that periodically checks for updates. Since Tiger has VFS improvements which allow apps to be notified when the filesystem changes, it is extremely likely that Spotlight takes advantage of this.

[Agent69]

Originally Posted by JLL

And mails, images, music, contacts?

Yes, all of it.

[Agent69]

Originally Posted by Hi I'm Ben

I think a part of people want to disable it to show off just how badass they are.

I don't need it, I don't want it. It is not about anything more than being able to turn off things I don't need. If you need it, fine.

[Silky Voice of The Gorn]

Originally Posted by Agent69

I don't need it, I don't want it. It is not about anything more than being able to turn off things I don't need. If you need it, fine.

I think we're all just dumbfounded that someone wouldn't want -or need- to search for something. Are you saying you have never used the "Find file" function on the Mac?

[Hi I'm Ben]

If you don't like spotlight just put a little piece of tape on the corner of the screen where the icon is, it doens't move

[ashtoash]

Let me put it better to you, indexing and spotlight IS NOT running as root. So it's subject to all the rules the user account you are logged into is.

[jasong]

Originally Posted by ashtoash

Let me put it better to you, indexing and spotlight IS NOT running as root. So it's subject to all the rules the user account you are logged into is.

Oh please, we all know you are just an Apple Apologist. Everyone knows that Spotlight is designed to read your every file and send the contents back to Steve Jobs personally.

It's actually in the EULA that you give Apple the right to use this information as Steve Jobs sees fit, this includes direct withdrawl from your bank account.

SPOTLIGHT IS EVIL!!!!!!!!!!!!!!!!!!!!!!!!!

[Silky Voice of The Gorn]

Originally Posted by jasong

Oh please, we all know you are just an Apple Apologist. Everyone knows that Spotlight is designed to read your every file and send the contents back to Steve Jobs personally.

It's actually in the EULA that you give Apple the right to use this information as Steve Jobs sees fit, this includes direct withdrawl from your bank account.

SPOTLIGHT IS EVIL!!!!!!!!!!!!!!!!!!!!!!!!!

[Peabo]

Originally Posted by Hi I'm Ben

Spotlight is neat. You will like it.

here is the spotlight pdf on apple's site

I think this is kinda stupid. You can't lock or hide this preference pane, so anyone who wants to see and find DIRECT LINKS to all the folders you don't want searched can just look here!

[JLL]

Originally Posted by z0ne81

I think this is kinda stupid. You can't lock or hide this preference pane, so anyone who wants to see and find DIRECT LINKS to all the folders you don't want searched can just look here!

Uhm, they have to be logged in as YOU!

[rmongold]

Originally Posted by JLL

Uhm, they have to be logged in as YOU!

Well, that and it's not meant to hide your documents from seaches other people may perform on your computer (necessarily, though it can be used that way if you let a bunch of people use the machine logged in as you), it's to let you specify what data/files you want to see in the return set. If you don't really care to see your MP3 collection show up in your Spotlight results, throw that folder in the exclude list. It's just a friendly way to help you prune the search results you see.

[leperkuhn]

I used to feel that I didn't really need a search tool, then I started using quicksilver. I've been using it more and more to open my documents, not just applications. And when I found out about spotlight I was psyched, because having it built into the system is going to be the greatest thing ever.

Instead of switching to the bulky finder, clicking applications, and scrolling for "itunes" you can just type apple-spacebar, then "itunes" and hit enter. That's it. I wasn't psyched for spotlight till i found out about the keyboard shortcut.

[EmmEff]

Mark me down as another OS X user who doesn't see any value in Spotlight... it is not a selling feature for me. And no, I've never used the Find File feature in Finder (ok, I did try it once and it didn't work as I was expected so I gave up).

I guess I just don't 'lose' files on my hard drive.

[Millennium]

Originally Posted by z0ne81

I think this is kinda stupid. You can't lock or hide this preference pane, so anyone who wants to see and find DIRECT LINKS to all the folders you don't want searched can just look here!

Do you honestly think that Apple wouldn't account for a hole this obvious? Their QA has been lacking as of late, but even Microsoft would catch that one.

[bewebste]

These security issues are cleary addressed in the Spotlight technology PDF from Apple's site (page 14):

Authenticated results

On systems with separate user accounts, Spotlight respects the boundaries of users� personal files�even though there is only one index per volume. When the Spotlight engine finds a set of results, it uses Mach messaging to communicate with the kernel and determine whether the current user has access to all of the data. Any files that the user isn�t permitted to see drop off the results list. Because Spotlight is tightly integrated with the file system and the kernel, this check takes no additional time. In fact, the whole process is transparent to the user. This mechanism works even if you�re applying access control lists (ACLs) to your user accounts.

To sum up, Spotlight is running as root, but when another process queries it for results, it will only return results the process has permissions for, so user A won't get results from user B's files when they do a search.

[Silky Voice of The Gorn]

Originally Posted by EmmEff

Mark me down as another OS X user who doesn't see any value in Spotlight... it is not a selling feature for me. And no, I've never used the Find File feature in Finder (ok, I did try it once and it didn't work as I was expected so I gave up).

I guess I just don't 'lose' files on my hard drive.

It's not about losing files. It's about getting to them quickly. Much quicker than drilling into a folder could ever be. Anyone who has used LaunchBar or ilk can tell you it utterly, completely changes the way you interact with your files and applications. And remember, Spotlight deals with metadata...an obscene amount of it. Combined with Smart Folders, you can do some extraordinarily powerful file management; conceptually, you wouldn't *need* to organize folders manually...

That said, Spotlight isn't 100% a Launchbar/QuickSilver killer, it doesn't have the smart-learning those apps have; in other words, every time you search for something its a "new" search. Typing "ps" won't get you to Photoshop. Spotlight is geared to file access, not app-launching (though it can be used that way, just not as efficiently as LB/QS)

[Silky Voice of The Gorn]

Originally Posted by z0ne81

I think this is kinda stupid. You can't lock or hide this preference pane, so anyone who wants to see and find DIRECT LINKS to all the folders you don't want searched can just look here!

To be pedantic, they can see only the folder name, not the path or "link" (whatever that means). Furthermore, once in that list, Spotlight won't return the folder at all, so little Johnny would have to manually search your ~/ to find the folder.

THAT said,I agree that that panel should have a lock on it (even though security is really only as good as the user..why is little Johnny able to access your account in the first place?)

[osxisfun]

Originally Posted by EmmEff

Mark me down as another OS X user who doesn't see any value in Spotlight... it is not a selling feature for me. And no, I've never used the Find File feature in Finder (ok, I did try it once and it didn't work as I was expected so I gave up).

I guess I just don't 'lose' files on my hard drive.

its about being able to search not only files but EVERY SINGLE WORD INSIDE OF EVERY SINGLE FILE...

that's a big difference. its like a whole new way of search that the vast majoirty have not even tried yet so we can not say its "useless" yet. Let's try this sucker out. I for one have already thought about how it would have help me over the last couple of weeks of alone.


I read some people here say they know where ALL their files are. Umm ok. good for you. But do you know what every single word in all those files that you say you know are?

Spotlight does.

[Hi I'm Ben]

Originally Posted by osxisfun

its about being able to search not only files but EVERY SINGLE WORD INSIDE OF EVERY SINGLE FILE...

that's a big difference. its like a whole new way of search that the vast majoirty have not even tried yet so we can not say its "useless" yet. Let's try this sucker out. I for one have already thought about how it would have help me over the last couple of weeks of alone.


I read some people here say they know where ALL their files are. Umm ok. good for you. But do you know what every single word in all those files that you say you know are?

Spotlight does.

i know where all my files are, what they are called. That's also not the point. The point is that I don't have to go through folders to find any of it. I can just type it in the little spotlight search. Boom, there it is. No navigating.

[osxisfun]

Originally Posted by Hi I'm Ben

i know where all my files are, what they are called. That's also not the point. The point is that I don't have to go through folders to find any of it. I can just type it in the little spotlight search. Boom, there it is. No navigating.

yah that is the major plus that does not get reported enough.


does anyone know if they spotlight SDK would allow for things like 3rd parties adding:

PS (launch photoshop)
M tom (mail tom)

etc....

[Silky Voice of The Gorn]

^ if it does, buh-bye LaunchBar...

[romeosc]

It is great it allows you to show everything that references to any word such as "baseball"

...... reveals pictures, songs, pdfs, text, files etc that have any reference to "baseball"

[macmike42]

First off, bear in mind I don't have a legal or illegal copy of Mac OS X 10.4.

I do, however, have a fairly intricate understanding of the HFS+ file system. It has always had this potential (Spotlight), but only now, and with the constant talk of the vaporware entitled "WinFS", has Apple decided to enable the existing capability, put it immediately to use, give it a public API, test and tweak it a little. Here is what is happening:

Spotlight is the name of an API which gives you access to a facility that has been available in HFS+ since its inception* . When you perform a Spotlight query, your are not querying a database server. You are querying the filesystem directly. The only separation between Spotlight and HFS+ are layers of abstraction which exist to allow access to the functionality (the API) and also to ensure that the "records" (which are really file system nodes such as files and directories) remain indexed properly so that they can be searched quickly.

Because the records that contain the metadata are the same objects as the files that you save data to and set permissions on, there is simply no security issue inherent in Spotlight, unless you have a local root exploit, but that would exploit everything, not just Spotlight.

If Apple actually made the index a physical file, visible on disk, then there would be possible security risks, but again, you need a root exploit and now you only get access to the index, not the data.

I'm not a security expert, but from an architectural point of view, Spotlight is as secure as the filesystem, because it is the filesystem.

* (You may be wondering why Classic Mac OS 8 or 9 didn't take any advantage of the capabillites of HFS+. This is simply because the kernel was incapable of doing so without a massive performance hit, and besides, Apple knew that the Classic Mac OS was dead by the time Mac OS 8.1 came out anyway.)

Finally, I would submit that he who does not feel the need to search their hard drive, probably doesn't do much with his computer.

[Silky Voice of The Gorn]

Macmike-

Thanks for that info, it's very informative. My only question is, Spotlight is indexing *something* (which you refer to as file system nodes), so are you saying that index resides not as some single, massive "file" but rather individual pointers to each file/folder?

[Person Man]

Originally Posted by Silky Voice of The Gorn

Macmike-

Thanks for that info, it's very informative. My only question is, Spotlight is indexing *something* (which you refer to as file system nodes), so are you saying that index resides not as some single, massive "file" but rather individual pointers to each file/folder?

Well, the metadata fields associated with each file that Spotlight uses have to be populated somehow when you first start using the OS.

[Silky Voice of The Gorn]

Originally Posted by Person Man

Well, the metadata fields associated with each file that Spotlight uses have to be populated somehow when you first start using the OS.

A file's individual metadata is already present when Spotlight indexes. I'm trying to clarify exactly what form this index takes. Macmike is saying (I think) that Spotlight is referencing the files directly, not looking up in a traditional database. Given that, I don't understand what exactly Spotlight is doing when it indexes, when it seems it doesn't need one to begin with...

[macmike42]

Sorry, I tried to be as detailed as I could, without taking up 6 pages.

A directory tree search will always be much slower than a search of a single, ordered index. But... some of the metadata might be created as Spotlight "indexes". For (a lame) example, the Finder currently creates and caches its own image thumbnails for images that don't already have them, but it doesn't share them with any other apps. These thumbnails could now be auto-generated metadata that gets populated at the same time that the index is updated (whenever the file is modified.)

For Spotlight to achieve the speed that (I assume) it has, it must use an index, just like any other database. Realize three things about a database index, as it relates to Spotlight: 1. the index does not have to contain the data itself, only a properly ordered "table of contents" since the file system nodes contain the actual data, 2. the index may or may not be an actual file, it may in fact be a special file system structure, and 3. the index, whether or not it is a physical file, may actually be more than one "file".

I think the main problem people are having understand this concept is that the metadata is stored within the filesystem nodes, while the index, which simply helps find this metadata quickly, must, by its very nature, be separate.

To paraphrase into one line: I have no idea what form the Spotlight index takes, or if such an entity even exists at all (although it is extremely likely that it does.)

This might be all wrong, but if it is, HFS+ is far more capable than I ever imagined (and than the available documentation claims it is.)

[Brass]

Originally Posted by macmike42

1. the index does not have to contain the data itself, only a properly ordered "table of contents" since the file system nodes contain the actual data

The index may not contain all the data in the files being index, but keep in mind that it MUST contain all the data that can be searched for. Otherwise, what is Spotlight looking up in the index when you type in a word to search for?

Having said that, it may not have the data (eg, the word being searched for) repeated in the index as many times as it is throughout all your files, but probably just once, with a pointer to each of the files that contains the word.

But technically, this means that the data IS in the index, albeit in a different form to how it is within the original files.

[macmike42]

Originally Posted by Brass

The index may not contain all the data in the files being index, but keep in mind that it MUST contain all the data that can be searched for. Otherwise, what is Spotlight looking up in the index when you type in a word to search for?

Having said that, it may not have the data (eg, the word being searched for) repeated in the index as many times as it is throughout all your files, but probably just once, with a pointer to each of the files that contains the word.

But technically, this means that the data IS in the index, albeit in a different form to how it is within the original files.

This is not necessarily true. The index may contain the actual words, or it may contain 3 characters from the beginning of the word, 3 from the middle, and 3 from the end, or (most likely) it may contain hashes of the words, or it may contain any combination of those, or something totally different that I haven't thought of. There are many ways to create an index, but if the index contains all the data that you could possibly search for, then it ceases to be an index and becomes the whole database.

[JCS]

Thanks for that info, it's very informative.

A lot of it is also wrong. Just one more week and the NDAs will be lifted...

[Silky Voice of The Gorn]

Originally Posted by JCS

A lot of it is also wrong. Just one more week and the NDAs will be lifted...

Ooh, I sense a smackdown
Yes, I suppose in a week it will be clearer how this all works.