[Le Flaneur]

Hi,

I've discovered someone on a MacWorld forum who found that Spotlight wasn't indexing his text files unless they had a .txt extension. Well, I'm finding that my Microsoft Word 2004 files are not being indexed by Spotlight *unless* they have a .doc extension! If, for example, I take one of the extension-less files and type:

2005-10-19 22:04:28.344 mdimport[316] Import '/Users/mbizer/Documents/Courses/Courses 05-06/FR320E Fall/revision pour partiel 1' type 'dyn.ah62d4rv4gk8zssccm2' no mdimporter

the error is "no mdimporter."

Here's what the mdls command reveals about 1) a W8BN file with a .doc extension, and 2) a W8BN file without the extension:

1) /Users/user/Documents/Courses/Courses 05-06/FR320E Fall/320E emploi du temps full.doc -------------
kMDItemAttributeChangeDate = 2005-10-19 17:40:52 -0500
kMDItemAuthors = (FR, ITL)
kMDItemContentCreationDate = 2005-09-25 18:39:08 -0500
kMDItemContentModificationDate = 2005-10-17 17:42:31 -0500
kMDItemContentType = "com.microsoft.word.doc"
kMDItemContentTypeTree = ("com.microsoft.word.doc", "public.data", "public.item")
kMDItemDisplayName = "320E emploi du temps full.doc"
kMDItemFSContentChangeDate = 2005-10-17 17:42:31 -0500
kMDItemFSCreationDate = 2005-09-25 18:39:08 -0500
kMDItemFSCreatorCode = 1297307460
kMDItemFSFinderFlags = 0
kMDItemFSInvisible = 0
kMDItemFSLabel = 0
kMDItemFSName = "320E emploi du temps full.doc"
kMDItemFSNodeCount = 0
kMDItemFSOwnerGroupID = 501
kMDItemFSOwnerUserID = 501
kMDItemFSSize = 42496
kMDItemFSTypeCode = 1463304782
kMDItemID = 2833114
kMDItemKind = "Microsoft Word document"
kMDItemLastUsedDate = 2005-10-17 17:42:31 -0500
kMDItemTitle = "320 emploi mwf"
kMDItemUsedDates = (2005-10-17 17:42:31 -0500)

2)
/Users/user/Documents/Courses/Courses 05-06/FR320E Fall/revision pour partiel 1 -------------
kMDItemAttributeChangeDate = 2005-10-19 18:52:47 -0500
kMDItemContentCreationDate = 2005-10-17 16:35:28 -0500
kMDItemContentModificationDate = 2005-10-17 18:24:43 -0500
kMDItemContentType = "dyn.ah62d4rv4gk8zssccm2"
kMDItemContentTypeTree = ("public.data", "public.item")
kMDItemDisplayName = "revision pour partiel 1"
kMDItemFSContentChangeDate = 2005-10-17 18:24:43 -0500
kMDItemFSCreationDate = 2005-10-17 16:35:28 -0500
kMDItemFSCreatorCode = 1297307460
kMDItemFSFinderFlags = 0
kMDItemFSInvisible = 0
kMDItemFSLabel = 0
kMDItemFSName = "revision pour partiel 1"
kMDItemFSNodeCount = 0
kMDItemFSOwnerGroupID = 501
kMDItemFSOwnerUserID = 501
kMDItemFSSize = 24064
kMDItemFSTypeCode = 1463304782
kMDItemID = 3075642
kMDItemKind = "Microsoft Word document"
kMDItemLastUsedDate = 2005-10-19 18:52:47 -0500
kMDItemUsedDates = (2005-10-17 18:24:43 -0500, 2005-10-18 19:00:00 -0500)

Note that while kMDItemContentType is different for the two files, the kMDItemFSCreatorCode and kMDItemFSTypeCode are the same.

It would then seem that the importer is looking at kMDItemContentType and not that the creator and type codes.

Why?

[Thinine]

OS X doesn't use creator and type codes. Never will. Though an equivalent may be developed. Don't count on it though.

[Le Flaneur]

People tell me that Spotlight is indexing their Word files even if those files don't have file extensions. Are you saying that this is not the case?

[CharlesS]

That's odd, according to the UTTypeTagSpecification in the MS Office spotlight importer's Info.plist file, it should be indexing files with type code "MSWD". However, I'm getting the same results you are.

[Thinine]

Maybe it's just broken. I don't know.

[tooki]

This shortcoming has been widely documented.

tooki

P.S. Tiger already introduced a really comprehensive file type/creator system at the OS level, but so far, nothing uses it.

[JKT]

I'm confused - what other file types from Word would be worth indexing (.dot perhaps)? Do you mean things like .rtf files created by Word?

[Le Flaneur]

My extensionless Word file produces this error when I ask Spotlight to import it:

2005-10-20 07:56:08.577 mdimport[209] Import '/Users/user/Documents/Courses/Courses 05-06/FR320E Fall/revision pour partiel 1' type 'dyn.ah62d4rv4gk8zssccm2' no mdimporter

The actual list of file identifiers is as follows:

Muse:~ user$ mdimport -r /Library/Spotlight/Microsoft\ Office.mdimporter/
2005-10-20 07:58:22.311 mdimport[216] Asking server to reimport files with UTIs: (
"com.microsoft.powerpoint.ppt",
"dyn.ah62d4rv4ge81a6dx",
"dyn.ah62d4rv4ge81u5db",
"dyn.ah62d4rv4ge81u5d1",
"com.microsoft.word.doc",
"dyn.ah62d4rv4ge81u5dx",
"dyn.ah62d4rv4ge81a6db",
"com.microsoft.excel.xls",
"dyn.ah62d4rv4ge81a6dy",
"dyn.ah62d4rv4ge80k55d"
)

So why isn't the UTI 'dyn.ah62d4rv4gk8zssccm2' included? What does 'dyn.ah62d4rv4gk8zssccm2' correspond to, exactly?

[Le Flaneur]

Originally Posted by tooki

This shortcoming has been widely documented.

Which shortcoming, and where has it been documented?

[Tsilou B.]

Hi,

the problem you and CharlesS are having is definitely no documented shortcoming of Mac OS X. I just created several Word documents without file extensions and they were all immediately indexed by Spotlight. mdls outputs:

kMDItemAttributeChangeDate = 2005-10-20 17:07:27 +0200
kMDItemAuthors = ("xxxxx")
kMDItemContentCreationDate = 2005-10-20 17:04:56 +0200
kMDItemContentModificationDate = 2005-10-20 17:04:56 +0200
kMDItemContentType = "com.microsoft.word.doc"
kMDItemContentTypeTree = (
"com.microsoft.word.doc",
"public.data",
"public.item",
"public.composite-content",
"public.content"
)
kMDItemDisplayName = "Test document"
kMDItemFSContentChangeDate = 2005-10-20 17:04:56 +0200
kMDItemFSCreationDate = 2005-10-20 17:04:56 +0200
kMDItemFSCreatorCode = 1297307460
kMDItemFSFinderFlags = 0
kMDItemFSInvisible = 0
kMDItemFSLabel = 0
kMDItemFSName = "Test document"
kMDItemFSNodeCount = 0
kMDItemFSOwnerGroupID = 0
kMDItemFSOwnerUserID = 501
kMDItemFSSize = 19456
kMDItemFSTypeCode = 1463304782
kMDItemID = 1521358
kMDItemKind = "Microsoft Word-Dokument"
kMDItemLastUsedDate = 2005-10-20 17:04:56 +0200
kMDItemTitle = "sampleword"
kMDItemUsedDates = (2005-10-20 17:04:56 +0200)

so Mac OS X recognizes the file as a Word document even though it has no extension. However, I have no idea why this doesn't work for you and CharlesS. It looks like there is a strange bug in Tiger that only affects some users.

[Le Flaneur]

Are you sure that the file extension isn't hidden?

[Tsilou B.]

Yes, I double-checked that. mdls also displays both names:

kMDItemDisplayName = "Test document"
kMDItemFSName = "Test document"

Only the display name would lack the hidden extension, the FSName should be the "real" name including any hidden file extensions.

[Hal Itosis]

Originally Posted by tooki

> This shortcoming has been widely documented.

> Tiger already introduced a really comprehensive file type/
> creator system at the OS level, but so far, nothing uses it.

I too would like to learn a bit more...

If you get a chance, could you toss out
a link (or two) for both statements.

Thanks.

[CharlesS]

Actually, I looked at an MS Word file, and it turns out that the actual correct type code for an MS Word file is W8BN, not MSWD. MSWD is actually the creator code. This would explain why the importer isn't recognizing Word files without a .doc extension.

[Le Flaneur]

Are you sure that the *type* code should be MSWD?

[Tsilou B.]

Originally Posted by Le Flaneur

Are you sure that the *type* code should be MSWD?

But Le Flaneur's file and my files have the same type and creator codes and the same extension (none). Nevertheless, my files are recognized as word files and indexed correctly and his files are not. There must be some other reason for this.

[CharlesS]

No, the type code shouldn't be MSWD. However, the Info.plist file for the MS Office importer has MSWD, initially fooling me into thinking that would be the type code for Word. However, it does not contain any reference to a word doc's actual type code, which is W8BN. W8BN does not appear anywhere in that Info.plist file that I can tell. You can look this up for yourself and verify it if you like (if W8BN does appear in yours, now that would be strange). So the importer doesn't know about the Word doc's type code, and the file has no extension, so how is the importer supposed to know that it is a Word file? This would seem to explain it IMO...

[Le Flaneur]

The question to which I'd like an answer is, "Why are we seeing this problem, while others are not?"

I'm checking with Microsoft, and a developer wrote,

I did set the creator to 'MSWD' in my own tests, but I just did the same tests setting the creator to nothing, and got the same results: mdls correctly recognizes all three file types, 'WDBN', 'W6BN' and 'W8BN' with a UTI of com.microsoft.word.doc.

The reverse case also fails: setting the creator to 'MSWD' and not setting the type correctly, and mdls gives a dynamic UTI.

[Tsilou B.]

Originally Posted by CharlesS

No, the type code shouldn't be MSWD. However, the Info.plist file for the MS Office importer has MSWD, initially fooling me into thinking that would be the type code for Word. However, it does not contain any reference to a word doc's actual type code, which is W8BN. W8BN does not appear anywhere in that Info.plist file that I can tell. You can look this up for yourself and verify it if you like (if W8BN does appear in yours, now that would be strange). So the importer doesn't know about the Word doc's type code, and the file has no extension, so how is the importer supposed to know that it is a Word file? This would seem to explain it IMO...

I believe I've found out how this works (at least on my Mac):

The Spotlight importer contains the UTI for Microsoft Word documents:

com.microsoft.word.doc

and Mac OS X itself declares that UTI. Here's a list of all system-defined UTIs:

http://developer.apple.com/documenta...section_1.html

The UTI includes the Type code W8BN and that's why Mac OS X recognizes the file as a "com.microsoft.word.doc" document and indexes it. If you look at the output of mdls on my Mac (see above), you'll see that Mac OS X has indeed recognized my Word file as a com.microsoft.word.doc file.



The question remains: Why does this not work on your Macs?

The file where all the default UTIs are saved is:

/System/Library/CoreServices/CoreTypes.bundle/Contents/Info.plist

In this file, there is an entry UTImportedTypeDeclarations, and the 16th subitem is the "com.microsoft.word.doc" entry that includes the "W8BN" type code.

Le Flaneur and CharlesS, is this entry present on your Macs, too?

[Tsilou B.]

If the entry is present on your Macs as well, then I suspect that you have another application installed on your computer that also declares the com.microsoft.word.doc document type and overwrites the system defined one. Do you have another word processor installed (maybe NeoOffice)?

[Tsilou B.]

You could try the following command in the Terminal:

find /Applications -name "Info.plist" -exec grep -l "com.microsoft.word.doc" {} \;

or

find /-name "Info.plist" -exec grep -l "com.microsoft.word.doc" {} \;

if you want to search the whole drive and not just the Application folder.

That should output the names of all files that could be the culprit. Open these files and look if they contain an UTImportedTypeDeclaration (or UTExportedTypeDeclaration) for com.microsoft.word.doc

[Le Flaneur]

Originally Posted by Tsilou B.

The question remains: Why does this not work on your Macs?

Good question!

Le Flaneur and CharlesS, is this entry present on your Macs, too?

Yes.

[Le Flaneur]

Originally Posted by Tsilou B.

Open these files and look if they contain an UTImportedTypeDeclaration (or UTExportedTypeDeclaration) for com.microsoft.word.doc

Well, I do have NisusWriterExpress 2.5. It doesn't have either of these type declarations, but within CFBundleDocumentTypes, the 4th entry has CFBundleTypeName "Microsoft Word Format," and under "CFBundleTypeOSTypes," entry 0 is "W8BN," and entry 1 is "W6BN."

So could that be causing problems?

[CharlesS]

CoreTypes.bundle is fine on mine and does include the W8BN type code.

I do have Fetch, Pages, NeoOffice, AbiWord, Pages Demo, and TextEdit on my hard drive, all of which register as being able to open W8BN files in their Info.plist. None of them mention com.microsoft.word.doc, though.

As for what mdls shows:

Code:
charles-imac:~/Desktop deleted!$ mdls test.doc
test.doc -------------
kMDItemAttributeChangeDate     = 2005-10-21 23:35:03 -0500
kMDItemAuthors                 = ("Charles S.")
kMDItemContentCreationDate     = 2005-10-21 23:35:02 -0500
kMDItemContentModificationDate = 2005-10-21 23:35:02 -0500
kMDItemContentType             = "com.microsoft.word.doc"
kMDItemContentTypeTree         = ("com.microsoft.word.doc", "public.data", "public.item")
kMDItemDisplayName             = "test.doc"
kMDItemFSContentChangeDate     = 2005-10-21 23:35:02 -0500
kMDItemFSCreationDate          = 2005-10-21 23:35:02 -0500
kMDItemFSCreatorCode           = 1297307460
kMDItemFSFinderFlags           = 0
kMDItemFSInvisible             = 0
kMDItemFSLabel                 = 0
kMDItemFSName                  = "test.doc"
kMDItemFSNodeCount             = 0
kMDItemFSOwnerGroupID          = 501
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 19456
kMDItemFSTypeCode              = 1463304782
kMDItemID                      = 2921586
kMDItemKind                    = "Microsoft Word document"
kMDItemLastUsedDate            = 2005-10-21 23:35:02 -0500
kMDItemTitle                   = "syzygy"
kMDItemUsedDates               = (2005-10-21 23:35:02 -0500)
charles-imac:~/Desktop deleted!$ mv test.doc test
charles-imac:~/Desktop deleted!$ mdls test
test -------------
kMDItemAttributeChangeDate     = 2005-10-21 23:35:16 -0500
kMDItemContentCreationDate     = 2005-10-21 23:35:02 -0500
kMDItemContentModificationDate = 2005-10-21 23:35:02 -0500
kMDItemContentType             = "dyn.ah62d4rv4gk8zssccm2"
kMDItemContentTypeTree         = ("public.data", "public.item")
kMDItemDisplayName             = "test"
kMDItemFSContentChangeDate     = 2005-10-21 23:35:02 -0500
kMDItemFSCreationDate          = 2005-10-21 23:35:02 -0500
kMDItemFSCreatorCode           = 1297307460
kMDItemFSFinderFlags           = 0
kMDItemFSInvisible             = 0
kMDItemFSLabel                 = 0
kMDItemFSName                  = "test"
kMDItemFSNodeCount             = 0
kMDItemFSOwnerGroupID          = 501
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 19456
kMDItemFSTypeCode              = 1463304782
kMDItemID                      = 2921586
kMDItemKind                    = "Microsoft Word document"
kMDItemLastUsedDate            = 2005-10-21 23:35:02 -0500
kMDItemUsedDates               = (2005-10-21 23:35:02 -0500)
charles-imac:~/Desktop deleted!$ GetFileInfo -t test
"W8BN"

I suppose this could just be a case of messed-up caches or something. I don't really care enough about this to go delete all the caches and set up all my default apps again, though. Someone else might want to try it...

[Tsilou B.]

Ok, I have created a small test application that queries the UTI database. You can download it from here:

http://www.savefile.com/files.php?fid=3464309

Run it and enter the UTI for Microsoft Word documents: com.microsoft.word.doc

On my Mac, the description is:

[FONT="Times New Roman"]Microsoft Word document[/FONT]

The location of the definition is:

[FONT="Times New Roman"]file://localhost/System/Library/CoreServices/CoreTypes.bundle/[/FONT]

and the definition is:

[FONT="Times New Roman"]{
UTTypeConformsTo = ("public.data", "public.composite-content");
UTTypeDescription = "Microsoft Word document";
UTTypeIdentifier = "com.microsoft.word.doc";
UTTypeTagSpecification = {
"com.apple.ostype" = W8BN;
"public.filename-extension" = doc;
"public.mime-type" = "application/msword";
};
}[/FONT]

So the UTI is defined by the system and does contain the W8BN typecode needed for identifying files without extension. What does the application output on your computers?

[Le Flaneur]

Originally Posted by CharlesS

I suppose this could just be a case of messed-up caches or something. I don't really care enough about this to go delete all the caches and set up all my default apps again, though. Someone else might want to try it...

Rebuilding (via the command lsregister) or deleting the LS cache won't help.

[Le Flaneur]

Originally Posted by Tsilou B.

You could try the following command in the Terminal:

find /Applications -name "Info.plist" -exec grep -l "com.microsoft.word.doc" {} \;

I ran this, but it didn't find anything. However, as noted above, I do have Nisus Writer Express on my hard drive.

[Le Flaneur]

Originally Posted by Tsilou B.

Ok, I have created a small test application that queries the UTI database. You can download it from here:

http://www.savefile.com/files.php?fid=3464309

Run it and enter the UTI for Microsoft Word documents: com.microsoft.word.doc

On my Mac, the description is:

[FONT="Times New Roman"]Microsoft Word document[/FONT]

The location of the definition is:

[FONT="Times New Roman"]file://localhost/System/Library/CoreServices/CoreTypes.bundle/[/FONT]

and the definition is:

[FONT="Times New Roman"]{
UTTypeConformsTo = ("public.data", "public.composite-content");
UTTypeDescription = "Microsoft Word document";
UTTypeIdentifier = "com.microsoft.word.doc";
UTTypeTagSpecification = {
"com.apple.ostype" = W8BN;
"public.filename-extension" = doc;
"public.mime-type" = "application/msword";
};
}[/FONT]

So the UTI is defined by the system and does contain the W8BN typecode needed for identifying files without extension. What does the application output on your computers?

When I run the app on my computer,

The location of the definition is:

file://localhost/Library/Spotlight/Microsoft%20Office.mdimporter/

and the definition is:

{
UTTypeConformsTo = ("public.data");
UTTypeDescription = "Microsoft Word Document";
UTTypeIdentifier = "com.microsoft.word.doc";
UTTypeTagSpecification = {"com.microsoft.word.ostype" = MSWD; "public.filename-extension" = (doc); };
}

Notice that unlike in the example above, com.apple.ostype is NOT defined!

[CharlesS]

Yep, and my results are identical to Le Flaneur's. Apparently the importer itself can override the Apple UTI specification. Weird.

[Le Flaneur]

Originally Posted by CharlesS

Yep, and my results are identical to Le Flaneur's. Apparently the importer itself can override the Apple UTI specification. Weird.

But can it be the importer that is doing this? Doesn't everyone have the same importer? I persist in thinking that it has something to do with the fact that we both have on our hard disks multiple apps that can read .doc files.

[CharlesS]

Yeah, but none of those apps care about com.microsoft.word.doc. And everyone who has both MS Word and TextEdit on their hard drive has multiple apps that can read .doc files...

[Tsilou B.]

I found out some more information. My Office spotlight plugin also includes the UTI definition without the com.apple.ostype (W8BN), so it looks like we all have the same plugin.
The Spotlight plugin with the broken UTI defines the UTI as "exported", the CoreTypes bundle defines the UTI as "imported". According to the Apple developer documentation, if both imported and exported UTIs are available, the exported UTI takes precedence over the imported one - in this case the spotlight plugin's UTI should be used instead of the CoreTypes one. This means that it's my Mac that does not behave according to the specifications.

So I went to the Library->Spotlight folder and dragged the Microsoft Office importer plugin to the desktop. Suddenly the OS noticed the UTI declaration in that file and the UTI Tester displayed the same information as on your Macs. With the spotlight plugin still on the desktop, I opened the Info.plist file inside with Property List Editor and simply deleted all these stupid broken UTExportedTypeDeclarations. After I had saved the file, the UTI Tester said that the com.microsoft.word.doc type was now undefined. I moved the spotlight plugin back to the Library->Spotlight folder, but that didn't help. I moved the file back to the desktop, copied it to the Spotlight plugins folder and deleted the original from the desktop. The UTI Tester then found the UTI inside the CoreTypes.bundle and everything was fine again.

I guess that this "solution" should work for you, too:

1.) Move the Office Spotlight plugin from Library->Spotlight to the desktop
2.) Show the package contents and open Info.plist with Property List Editor or TextEdit.
3.) Delete the UTExportedTypeDeclarations

If you only have TextEdit, delete all lines between

Code:
<key>UTExportedTypeDeclarations</key>

and

Code:
<key>CFBundleDocumentTypes</key>

and the first of these lines itself:

Code:
<key>UTExportedTypeDeclarations</key>

4.) Save the file
5.) Copy (don't move) the Spotlight plugin back to the Library->Spotlight folder
6.) Delete the plugin from the desktop by putting it into the trash and emptying the trash.

Use the UTI Tester to check if com.microsoft.word.doc is now correctly found in CoreTypes.bundle. If it is, reindex your hard drive and all your Word files should finally be indexed.

[CharlesS]

Yep, that did the trick. Congrats to you, Tsilou B.

[Le Flaneur]

Originally Posted by Tsilou B.

I guess that this "solution" should work for you, too:

1.) Move the Office Spotlight plugin from Library->Spotlight to the desktop
2.) Show the package contents and open Info.plist with Property List Editor or TextEdit.
3.) Delete the UTExportedTypeDeclarations

If you only have TextEdit, delete all lines between

Code:
<key>UTExportedTypeDeclarations</key>

and

Code:
<key>CFBundleDocumentTypes</key>

and the first of these lines itself:

Code:
<key>UTExportedTypeDeclarations</key>

4.) Save the file
5.) Copy (don't move) the Spotlight plugin back to the Library->Spotlight folder
6.) Delete the plugin from the desktop by putting it into the trash and emptying the trash.

Use the UTI Tester to check if com.microsoft.word.doc is now correctly found in CoreTypes.bundle. If it is, reindex your hard drive and all your Word files should finally be indexed.

You are one smart guy. Thanks so much! I would add one thing. After following the steps above, type in Terminal:

mdimport -r /Library/Spotlight/Microsoft\ Office.mdimporter/

in order to reimport all those neglected files without the .doc extension.

[Le Flaneur]

The people at Microsoft are telling us that we shouldn't remove the UTExportedTypeDeclarations
tags from the Office importer:

"I would strongly advise against removing the UTExportedTypeDeclarations
tags from the Office importer since if his Launch Services database gets
rebuilt, nothing will make Spotlight pass Office documents to the Office
importer.

The customer should restore the importer and if the same problem occurs,
enter:

/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Fram
eworks/LaunchServices.framework/Versions/A/Support/lsregister -f
/Library/Spotlight/Microsoft Office.mdimporter"

Surely this is wrong, isn't it?

[CharlesS]

Hmm, com.microsoft.word.doc still is in the LSItemContentTypes, even if you've deleted the UTExportedTypeDeclarations, so I would think it would still know to hand that type to the importer. If you really want to know for sure, though, you could go and delete the LaunchServices database and see what happens.

Alternatively, you could change UTExportedTypeDeclarations to UTImportedTypeDeclarations so they wouldn't override the one in /System. This is what Apple seems to do with most of their importers. However, SourceCode.mdimporter seems to function just fine without any type declarations...

[Tsilou B.]

I have just tried deleting the LaunchServices database after I had deleted the UTExportedTypeDeclarations, and the Mac had absolutely no problems indexing Word files thereafter. Maybe you can tell the people at Microsoft that the Office UTIs are already defined by Mac OS X itself and if they want to include the UTI in their importer as well, then they should not forget defining the type code (W8BN).

[CharlesS]

This is Microsoft we're talking about, though... do you really expect them to do something right?