There are two basic ways to do it. One is to define a generic class for plugin makers to extend by subclassing, much like how the NSView architecture in Cocoa itself works. Another would be to define a formal protocol describing what methods plugin objects must implement. You can even do both. Either way, you don't necessarily need to include a framework in your app and define the class/protocol there; it can be defined in the app executable itself, and you could distribute just a header file to plugin developers.
There's another part to the task of creating a plugin architecture, though, and that's figuring out when to use the plugin objects. If you've got several subclasses of (for example) ImageExportFilter, you'll need to map human-readable names to them and include them in your Export panel. If you've got subclasses of SoundDecoder, you'll need to know what subclass to instantiate and use for a given sound format.
For examples of how a Cocoa-based plugin architecture could work, see Omni's
open-source frameworks. The OmniFoundation framework defines a "bundle registry" system which can be used for automatically loading bundles and keeping track of loaded bundles. You can use it to help figure out when to use this or that plugin object. Other parts of our framworks show plugin architectures in action -- the OAPreferences subsystem in OmniAppKit is a plugin architecture for preference panes, and OWF and OmniHTML are full of modular systems.