[gorickey]

Application Enhancer Isn't Bad

Amen.

[CharlesS]

A lot of the arguments are along the lines of "That's not APE's fault, it's the modules." Yes, but by the nature of what an APE module does, it makes it likely to cause application crashes and strange behavior. You could use the same logic and say that OS 9's extension mechanism wasn't bad because it was the individual extensions that were causing the problems.

[Macanoid]

I've been running windowshade, fruit menu, shapeshifter, ape and dock detox the moment these haxies came out and can honestly say I have never had any problems whatsoever. Great stuff. My Mac would feel VERY different without them.

[philzilla]

even when i used to beta test Slava's "extra high risk alpha versions" i never had much problem. i really don't know what some people get up to with their version of OS X to make it so unstable. i guess it's just real easy to point the finger at Unsanity.

[dialo]

I noticed the same thing regarding the crash logs and wondered if this was why people thought APE was to blame. However, I've never cared that much, just assumed everyone knew something about APE that I didn't and all of the claims of crashing were related to something totally different that I hadn't experienced.

[superfula]

Originally posted by philzilla:
even when i used to beta test Slava's "extra high risk alpha versions" i never had much problem. i really don't know what some people get up to with their version of OS X to make it so unstable. i guess it's just real easy to point the finger at Unsanity.

Well when you start hacking up your OS, there really isn't much else to blame aside from the hack. Just like any softare, and even an OS, some will have problems and others don't. No two systems are alike.

[Mike S.]

Very informative write-up, thanks for making it.

I tend to disable APE and log out before applying a system update or security patch (those that require a restart anyway). Is this at all necessary or can the modules have no effect on the update process?

[gorickey]

Originally posted by Mike S.:
Is this at all necessary or can the modules have no effect on the update process?

I've never had a problem and have never disabled APE on any update...

[mitchell_pgh]

Originally posted by gorickey:
I've never had a problem and have never disabled APE on any update...

I used APE for a while but have since removed it.

WHY? No reason.

[Arkham_c]

It may be the case that the framework is not the problem, but if you cannot use modules that utilize the framework without problems, what's the point?

I subscribe to the RealBasic developers beta list, and I have seen dozens of programmers who have had problems reported by users that were ultimately traced to one APE module or another. I just refuse to support APE with my apps -- if a user has a problem, I tell them to remove APE, and the problem goes away.

So, I still don't like APE and cannot recommend it to anyone, even if the cause is in the modules.

[JKT]

Given that WindowShade X caused problems for me and many others back in the upgrade from 10.1.x to 10.2 days and also that MacFixIt.com often cite APE and haxies as a commonality in a lot of the esoteric problems people get during system upgrades etc. I dispute that these things are not a problem or, at the least, a potential problem.

If you use them, then fine, but people should be made fully aware that they are the first port of call when you start suffering any weirdness with the system... and that they definitely should be disabled whenever you upgrade anything on your system.

Edit: FWIW, when using 10.1.x, WindowShade X also caused several issues with OmniWeb and minimised windows for me too (they wouldn't re-maximise from the Dock or would be blank when they did so and would cause OW to freeze). I never realised it at the time, but it wasn't until I ditched WS X after the 10.2 problem that it dawned on me that this was the cause.

[JKT]

Originally posted by Mike S.:
Very informative write-up, thanks for making it.

I tend to disable APE and log out before applying a system update or security patch (those that require a restart anyway). Is this at all necessary or can the modules have no effect on the update process?

Read the Windowshade X thread, and decide for yourself.

[gorickey]

Originally posted by JKT:
... and that they definitely should be disabled whenever you upgrade anything on your system.

Not here.

[iNeusch]

Originally posted by gorickey:
Not here.

Same for me, the APE has been made the smart way and it is safe, even when upgrading !

[octane]

Some of you might find this /. article interesting, especially the responses to Unsanity's posting of their defence of APE.

http://apple.slashdot.org/article.pl.../05/27/2351237

[Basilisk]

One thing that folks seem to forget is that APE is not the only patching technology available. I've seen folks post their dislike of APE before and in different thread praise Codetek Desktop, Desktop Manager, etc.

These are all based on mach_inject/mach_override which is just as dangerous (arguably more dangerous) than APE.

Here's a recent list of mach_inject apps, if you are condemning APE you should be avoiding these too.

http://rentzsch.com/mach_inject/mach_Users

Alex

[GENERAL_SMILEY]

I use APE, I really don't have a problem with it - but occasionally Safari quits unexpectedly - and the crash report always references the APE module (and launchservices too). Now I don't know if thats even relevant, or what it means.

I have detour, audio hijack, paranoid android, and shapeshifter - I have disabled everything but the android, but it still happens.

[Mike S.]

Originally posted by JKT:
Read the Windowshade X thread, and decide for yourself.

I read it but nothing there really seems to indicate that it would interfere with the process of updating.

All I got from that thread was that Windowshade X was causing general performance degradation due to a bug. That's certainly not a good thing but reduced performance is hardly a catastrophe and disabling it restored regular performance.

I also find it interesting that the thread was from 2002 and APE was a "new format", is that the most recent wide spread issue attributed to Unsanity products? If so, that's a pretty good track record in my book.

Thanks for the link.

I still wouldn't mind hearing an answer to my original question from the APE developers; can APE Modules (I currently only have PA), in theory, mess with an update process (i.e.. cause it to fail, leading to system instability or a broken system install) or is it coded in such a way that it simply can not happen?

[CharlesS]

Originally posted by Mike S.:
I still wouldn't mind hearing an answer to my original question from the APE developers; can APE Modules (I currently only have PA), in theory, mess with an update process (i.e.. cause it to fail, leading to system instability or a broken system install) or is it coded in such a way that it simply can not happen?

Theoretically it can mess with any application running under your user space.

[gorickey]

Originally posted by CharlesS:
Theoretically it can mess with any application running under your user space.

Much like anything can "theoretically"...

[Adam Betts]

I'm surprised anyone haven't recognized the thread title. It's brilliant, gorickey

"Are you scared?"

"Yes!"

"You shouldn't be.."

"Huh?"

[Person Man]

Originally posted by Arkham_c:
It may be the case that the framework is not the problem, but if you cannot use modules that utilize the framework without problems, what's the point?

I subscribe to the RealBasic developers beta list, and I have seen dozens of programmers who have had problems reported by users that were ultimately traced to one APE module or another. I just refuse to support APE with my apps -- if a user has a problem, I tell them to remove APE, and the problem goes away.

So, I still don't like APE and cannot recommend it to anyone, even if the cause is in the modules.

Why tell people to disable APE entirely to use your software? Have you ever considered having them add your application to APE's exclude list and seeing if the problem goes away? That's the problem I have with developers telling people to get rid of APE...

"So APE is causing my application to crash, huh? APE is bad, get rid of it entirely."

How hard is it to tell people first to add your application to APE's exclude list and see if the problem still exists? If after doing that, the problem still isn't fixed, THEN tell them to turn APE off.

One way that APE is different from OS 9 extensions is that OS 9 extensions didn't have a way you could turn them off for an individual application the way APE lets you.

Seriously, is it that much harder for you to ask users to try disabling APE *only for YOUR application* before telling them to get rid of it entirely?

You still don't have to support APE with your applications. But at least you can ask people to try adding your application to the exclude list before telling them to get rid of it entirely.

[CharlesS]

Originally posted by gorickey:
Much like anything can "theoretically"...

Okay, let me state this a little more strongly. APEs such as Windowshade and Demetallifizer which add features to every window on the system do mess with every application (with GUI, at least) which is running in your user space.

[yukon]

I'm agreed with CharlesS and Arkham_c, APE is generally a bad throwback to OS9, although it's not quite as bad as APE modules (apparently) don't patch the system, rather the applications (if it patches things like loginwindow, I'll change that opinion). The APE framework isn't a horrible thing, but it could be misused, and not just intentionally misused. APE breaks through memory protection from application to application. The modules may cause applications to crash, which is a fault of the module, but APE is what's allowing the bad behaviour to happen. There's a comment on the slashdot page, about how it's against UNIX principles (the author repeats "Hands off the drivers!"), so I suppose it's against the new MacOS principles as well.

I've had trouble with APE in 10.2, and despite the author's denying it, traced it to some sort of conflict between MenuCracker and APE. Removing all modules wouldn't fix the problem, removing APE fixed the problem. The finder would launch, then crash, then try to relaunch forever. Maybe this was a really odd thing to happen, but APE does things it's not supposed to, it wasn't the Finder's fault that it was crashing. I've since removed all hacks, and things are going fine.

Apple should just give up on the consistant-ui thing, obviously it's being circumvented and causing even more trouble, a clean and easy way to theme or make UI modifications, even if it's hidden, would be fine.

[Chuckit]

Originally posted by yukon:
APE breaks through memory protection from application to application. The modules may cause applications to crash, which is a fault of the module, but APE is what's allowing the bad behaviour to happen. There's a comment on the slashdot page, about how it's against UNIX principles (the author repeats "Hands off the drivers!"), so I suppose it's against the new MacOS principles as well.

APE doesn't really break memory protection any more than plug-ins (such as QuickTime codecs or CMMs) do.

Originally posted by yukon:
I've had trouble with APE in 10.2, and despite the author's denying it, traced it to some sort of conflict between MenuCracker and APE. Removing all modules wouldn't fix the problem, removing APE fixed the problem. The finder would launch, then crash, then try to relaunch forever. Maybe this was a really odd thing to happen, but APE does things it's not supposed to, it wasn't the Finder's fault that it was crashing. I've since removed all hacks, and things are going fine.

So I suppose this could just as easily be a cautionary tale about MenuCracker?

[yukon]

APE doesn't really break memory protection any more than plug-ins (such as QuickTime codecs or CMMs) do

Well, codecs and CMMs are used for specific things, they're incorporated into the program, rather than just patched into memory when the program is launched. The author said it himself, there's no guarantee when the CMM will be loaded etc, so it can't be used for the type of modifications they're after..."injecting" into memory is a bruteforce way to change anything, and it has it's problems.

AFAICT, the author meant "memory protection" in a "applications messing with system" kind of MacOS7 way, whereas the term taken as applied in full would mean "Applications don't mess with the memory areas of other processes". I'm not sure of the exact definitions of the terms Apple would use, but since the system itself is made up of processes, it seems that hacking that would un-protect memory. An easy example is where an application would crash horribly and scramble the memory of another app, you can watch that happen in OS9, but it's supposed to be impossible in OS X. If there's a "crash" in a "module", or just some bad programming, every application who's memory get's patched suddenly gets scrambled.

just as easily be a cautionary tale about MenuCracker?

Basically, yeah. Both patched things, they didn't like each other. They conflicted. That's the whole point, isn't it? There can be patches which conflict with each other, patching the same routine or area in memory, causing problems.

[CharlesS]

Originally posted by Chuckit:
APE doesn't really break memory protection any more than plug-ins (such as QuickTime codecs or CMMs) do.

No, this is just one of these old wives' tales that manages to circulate amongst the general populace. Kind of similar to a rumor I've seen floating around the boards that claims BootCD boots slowly because it loads the whole OS onto a RAM disk (WTF? How could I possibly do that with a RAM disk that's only 10 MB?! )

Anyway, back on topic: Here's the truth behind this claim. With a plug-in like a QuickTime codec, the author of the host app has deliberately coded his app to accept plug-ins. The author has taken into consideration the things that the plug-ins might do and has coded the rest of the app appropriately. Also, there are often limitations on what plug-ins can do, how many things they can mess with, and usually there's enough separation between them that they can't really conflict with each other.

Now, let's look at APEs. They can mess with any app, regardless of whether the author has written his app to handle plug-ins or not. Consequently, if the author makes assumptions about what some piece of code is going to do, and then a haxie screws with that code, you can cause weird behavior in the app. Now, throw in two different haxies that try to patch the same code, and things get really fun - anyone remember OS 9 extension conflicts? Well, that's exactly the same situation as what is going on here.

Memory protection means that apps don't screw with each other. Apps can still support plug-ins if they intentionally do so. Adding a plug-in is like me inviting you into my house. A haxie is like you breaking my window and entering without permission.

So I suppose this could just as easily be a cautionary tale about MenuCracker?

I don't know much about how MenuCracker works, but I'm guessing it probably patches things just like APE does. So this is basically a description of an OS X extension conflict. At least MenuCracker only patches one area (the code that blocks third-party menu extras) but it's still not such a safe thing. It's too bad Apple won't just open up the API so people wouldn't have to use this nonsense - it's not as if the menu bar space is sacred with the current setup, since any app can just use an NSStatusItem anyway - it's just harder to remove a menu item added in that manner.

[goMac]

Actually Apple usually provides sensible alternatives to most stuff people reverse engineer. Login plugins is one example.

There is also no reason you can't launch your own app in the background and run a menu extra. You could configure a nice, previously mentioned, login plugin to do this for you. You don't have to use the login plugin API.

I like a quote I got from someone who works at Apple: If OS X doesn't let you do it, there is probably a good reason why. From a coding standpoint this is correct. If you are unable to code something using bare Apple API's, it's probably not a good idea to code it, especially when Apple provides perfectly good alternatives. Apple doesn't close API's without rhyme or reason. Apple closed menu extras for a reason, but there are still other ways of coding them. The same goes for a lot of stuff on OS X that people seem to want to find a way to code around.

Sorry if I make little sense right now, I'm in dire need of sleep...

[Mike S.]

Interesting commentary.

Now let me pose another question. Shapeshifter is really the premiere APE Module and you've established that "code injection" isn't the best thing to do, etc.

However, is this method any different than the way Kaleidoscope or Windowblinds work?

Somebody had also mentioned something called mach_inject (perhaps it was at Slashdot) so it seems that there are system level ways to offer code injections. If this is such a bad thing then why are there supported ways to do it?

Furthermore, while the method is frowned upon from a philosophical point of view; if people aren't having any wide spread trouble with it then is it really so bad in practice?

[Basilisk]

I'm sad to say that this post got quoted back out of context into the Unsanity blog comments, and in fact has become the "old wives tale" that I hoped I was correcting. Please see my comments here http://forums.macnn.com/showthread.p...75#post2015375 for a followup when some of this thread was taken out of context by a user.

Anyway, back on topic: Here's the truth behind this claim. With a plug-in like a QuickTime codec, the author of the host app has deliberately coded his app to accept plug-ins. The author has taken into consideration the things that the plug-ins might do and has coded the rest of the app appropriately. Also, there are often limitations on what plug-ins can do, how many things they can mess with, and usually there's enough separation between them that they can't really conflict with each other.

Absolutely, patently false.

As a simple example, every single Cocoa application loads InputManager bundles. This is precisely how PithHelmet uses SIMBL to load itself in Safari. By default Cocoa applications have no direct control over this load, and AFAIK can't prevent it even if they want to.

You may believe that APEs are a different beast, but claiming that apps control all the plugins they load is false, and spreading tha claim precisely the kind of misinformation you lamented in your own post.

APE breaks through memory protection from application to application.

No, it doesn't. I understand that this is a simplification that folks are using, but its not true except in the grossest terms and its been repeated so often that people have forgotten the real explanation of this.

Memory protection isn't "broken" by APE, it never existed in the absolute sense that folks seem to believe. A system that didn't allow processes to share memory would be useless. They could never communicate. You couldn't interact with them because they would have no way to receive events from the hardware.

There are documented, well-defined ways to pass or share memory between processes in MacOS X. One example is mach ports, and the various structures built on them (NSConnection, Distributed Objects, etc.). These are a feature of the OS and they are used pervasively throughout the system.

All too often in the recent rash of APE discussions people claim that these mechanisms are for debuggers only, but that's simply untrue. Everytime you change a hardware setting, everytime an keystroke event from your keyboard leaves the kernel, mach ports made that happen.

APE uses that same mach port mechanism to load code into other applications. You can argue that using that mechanism to patch code is potentially risky and I would agree (so long as you said "potentially"). However, memory protection wasn't violated any more than the kernel passing keyboard events violates memory protection. Both required different processes to share memory.

What's more, there are several other ways besides APE that applications can get inside other processes. APE is simply the most visible because its popular with developers. Ironically its popular becuase in fact is _safer_ than a lot of the other alternatives. Mach inject does the same thing, but its much harder to use, and developers have to be much smarter and more careful when using it.

don't know much about how MenuCracker works, but I'm guessing it probably patches things just like APE does.

Not except in the broadest terms, both patch code. MenuCracker loads into SystemUIServer by pretending to be an Apple menu extra (a plugin that SystemUIServer allows to load). Once there it patches the selector for the Apple security mechanism to prevent it from blocking other extras.

Note that I said it loads only in SystemUIServer. It can't load at all in the Finder (the Finder doesn't have a mechanism to load NSMenuExtra classes at all).

APE has its risks to be sure. But please try to be factual in your criticism of it.

Alex

[eevyl]

You know, if that article was not from the guy who works on the company that makes APE things to make money for, it would be a lot more interesting ...

I do not use APE, not just because they can cause instability, but because I didn't find worth the money nor the risk for myself. I see the need for it though, I was a happy Kaleidoscope user/schemer in the ol' days.

[milhouse]

I really love the functionality of Usanity's products.

However, without fail, when I use them, I'll eventually have crashes or system freezes. If I disable APE and the associated apps, my problems seem to go away. (Sometimes these repeated crashes have resulted in corruption requiring a reinstall.)
The problems don't seem to appear for weeks or months and while many "things" can occur in that time frame, disabling APE seems to be a "fix".

APE et al may not be the primary culprits but there is some correlation with their use, at least in my experience.

I wish I had a clue about deciphering crash logs...

[goMac]

I took a look at the mach_inject vs. APE commentary and there is one interesting mis-conception I noted.

Mach_inject is probably in most cases safer than APE. Why? APE and mach_inject are completely different tools, even though they basically do the same thing. They are built different however. Mach_inject is basically the equivalent of surgeon's knife while APE is some sort of large weapon of mass destruction.

APE contains something very important that mach_inject does not, a daemon. Injecting code has to always be done on a per-application basis. The APE daemon takes APE modules, and injects them into every application that ever launches (unless it is excluded).

Mach_inject, on the other hand, takes code and only injects it into the programs the programmer tells it to, from a running program. It is a library for use in a program, not an actual program or bundle manager. Therefore it is much safer. For example, a program integrating CVS into the Finder (which was on the list of mach_inject using programs) does not patch Safari, iChat, iTunes, System Preferences, etc. The program is built to just patch the Finder. This makes it a lot safer than APE because it is only patching exactly what it needs to.

The CVS example I've never tried myself, so I hope for the sake of my argument it is only patching the Finder. But from the work I have done with mach_inject these are the differences between it and APE.

[goMac]

Originally posted by Mike S.:
Somebody had also mentioned something called mach_inject (perhaps it was at Slashdot) so it seems that there are system level ways to offer code injections. If this is such a bad thing then why are there supported ways to do it?

Mach_inject is a 3rd party library mentioned earlier. It is NOT an Apple supplied library. I've written about it in my last post.

[iOliverC]

Originally posted by goMac:
I took a look at the mach_inject vs. APE commentary and there is one interesting mis-conception I noted.

Mach_inject is probably in most cases safer than APE. Why? APE and mach_inject are completely different tools, even though they basically do the same thing. They are built different however. Mach_inject is basically the equivalent of surgeon's knife while APE is some sort of large weapon of mass destruction.

APE contains something very important that mach_inject does not, a daemon. Injecting code has to always be done on a per-application basis. The APE daemon takes APE modules, and injects them into every application that ever launches (unless it is excluded).

Mach_inject, on the other hand, takes code and only injects it into the programs the programmer tells it to, from a running program. It is a library for use in a program, not an actual program or bundle manager. Therefore it is much safer. For example, a program integrating CVS into the Finder (which was on the list of mach_inject using programs) does not patch Safari, iChat, iTunes, System Preferences, etc. The program is built to just patch the Finder. This makes it a lot safer than APE because it is only patching exactly what it needs to.

The CVS example I've never tried myself, so I hope for the sake of my argument it is only patching the Finder. But from the work I have done with mach_inject these are the differences between it and APE.

I thought for Xtender you were patching every application, just like APE?

[Basilisk]

APE contains something very important that mach_inject does not, a daemon. Injecting code has to always be done on a per-application basis. The APE daemon takes APE modules, and injects them into every application that ever launches (unless it is excluded).

APE injects code only into processes that match the requested bundle information. For example, I myself offer a APE module that injects only into the Dock (PullTab).

I understand that what you really mean is that many (if not most) APE modules request to be loaded into every application bundle, but this is not a unique feature. mach_inject can be used for the same purpose, as demonstrated by DefaultFolder X and Quickeys, both are mach_inject based and both inject into every running application except those specifically excluded.

This makes it a lot safer than APE because it is only patching exactly what it needs to.

Again, APE has precisely this functionality, APE modules can and do select specific apps they are intended to run in.

APE is in fact safer than mach_inject because it uses a patch table that lets patches be unwound and appropriately stacked. This is a critical feature, and it makes APE a much safer and better tool.

Let's get technical for a moment:

When an APE module patches a system function what really happens is that the APE framework places a generic call in place of the original function. When the application uses the patched system call the call is redirected to a table of current overrides, which the APE framework then uses to locate and redirect to one or more patches. This extra level of indirection has two important consequences:

- It means that APE modules can "unpatch" system calls once they have done their work. What is left is an empty table which APE then redirects back to the original system function.

- It means that multiple APE modules loaded in a single application can safely patch the _same_ system function with considerably less risk. Becuase all patches are registered with the APE framework first, the framework can mediate between the conflicting requests and stack the patches appropriately.

The importance of these two advantages cannot be overstated. One of the major problems with OS 9 extensions was that because there was no standardized way to accomplish such patching frequently extensions used naive or inappropriate assumptions when patching system calls. OS 9 extensions authors has to decide between head versus tail patching, and be very careful to handle patch stacking appropriately. Here are some typical examples of the kindof complexities involved: http://groups.google.com/groups?selm...162.219&rnum=4

When folks talk about extension conflicts on MacOS 9 what many users never understood was that conflicts were often the result of bad patch stacking. A patch which had a subtle incorrect assumption in its patch mechanism might be fine alone. It was when it was then combined with another extension's patch code with a diffent patch mechanism that the problem occurred.

APE helps eliminate many of these problems because it centralizes the patch mechanism. In other words it has all the same advanages of any other MacOS X framework. It takes much of the complexity burden away from developers and places it in one centralized place that can be updated and fixed without requiring every developer that uses the framework to recompile their application. Tomorrow if a serious problem is discovered in APE's patch mechanism Unsanity can fix the bug and release a new framework and I don't have to recompile PullTab.

Additionally APE can be much smarter than MacOS 9 extensions becuase it "sees" all the patches as a whole. It can resolve some types of problems simply because of this increased visibility. MacOS 9 extensions had no way to do this and we all suffered from the result.


Patch unwind and patch stacking are so important that plans are underway to add them to mach_inject. Technical users might be interested in reading ExtendaMac (http://extendamac.sourceforge.net/). In particular check out recent posts on the mailing list (http://sourceforge.net/mailarchive/f...endamac-macosx) where this has been recently discussed.


APE is not perfect, and inappropriately used can be risky. But mach_inject is just as risky, and for a certain category of problems, much less safe.

Hope ths helps clear some things up...

Alex

[Will McGoonigle]

I've been waiting 9 months for the fricking Labels X update.

[goMac]

Originally posted by Basilisk:
- It means that APE modules can "unpatch" system calls once they have done their work. What is left is an empty table which APE then redirects back to the original system function.

- It means that multiple APE modules loaded in a single application can safely patch the _same_ system function with considerably less risk. Becuase all patches are registered with the APE framework first, the framework can mediate between the conflicting requests and stack the patches appropriately.

Mach_inject can do the same thing quite simply. You just keep a reference to the original pointer to the function. You can continue calling the original function from your patch, or simply "disable" your patch and make the default behavior the original function.

I'm not entirely sure, but I think it's impossible to remove code after it has been inserted. No example I have seen has ever done this. Rather you disable your patch using the method above. Your patch is still in place, but just redirects to the default pointer.

This also means mach_inject can also do stacking. Lets say I load MachWindowShade which overrides the minimize behavior. This places in its own minimize functionality. The programmer was lazy and he didn't want to allow the user to use default minimize behavior so he didn't add a pointer to the original function.

Now another programmer comes along and writes MachBeepOnMinimize (which he has blatantly ripped off the Beep On Window Close example). Because his program is simply adding functionality, he does add a pointer to the original function. But, from what his program can tell, MachWindowShade's function IS the original function. So, it stores a pointer to MachWindowShade's function. When the beep function is called, it beeps, then calls its "original" pointer which calls MachWindowShade. The two mods have just stacked.

Granted this isn't built in to mach_inject, but it is usually used. The beep example uses something similar to continue using the close method while overriding it.

APE is not actually removing the code it has put into place. Rather it sounds like it installs manager code into an application and handles pointers from there. You simply can't "remove" patch code.

Anyway, don't take my word as the bible. I didn't write our application enhancer, although I know mostly how it works. I've never seen the APE source either obviously.

[Basilisk]

Mach_inject can do the same thing quite simply. You just keep a reference to the original pointer to the function. You can continue calling the original function from your patch, or simply "disable" your patch and make the default behavior the original function.

I think you missed the point of my post. I'm aware that it is technically possible to build APE-like patch table/jump island functionality into mach_override. The point I'm making is that this is not actually built-in to the library. Each developer is responsible for implementing this functionality themselves. And if the extension problems on MacOS 9 are any indication, well-intentioned, smart developers will still sometimes screw this up.

Again, I'm hardly out in left field with criticism of mach_override. The need for this missing functionality was acknowledged by its author on the Extendamac mailing list just _last week_.

Granted this isn't built in to mach_inject, but it is usually used. The beep example uses something similar to continue using the close method while overriding it.

We were talking about risks. I'm aware that its possible to do this with mach_override. However, the fundamental question is it better to have multiple authors implement the same thing over and over (patch redirection with jump island) or use a single well-understood framework for the same purpose? I would argue that the history of trap patching on OS 9 is a pretty clear indicator that smart people still make mistakes. A framework approach is the "Right Thing", it centralizes the hard part of the code and makes upgrades easier.


In any case, I don't think your post addresses my core reponse to your original post. Both APE and mach_override do the same thing in very similar ways. Both can be used to inject code into every running app or just one application. Thus both are equally risky in potential widespread system effects, and mach_inject is not a "surgeon's knife" compared to APE. If Duality is doing theme changes via mach_inject then you must be patching all running apps, right?

APE is safer today becuase it offers a simple, standard way to handle jump islands for patches in a thread-safe manner. This is why I chose it for PullTab, and its why other developers have chosen it for their enhancements. mach_override can be used to implement these same features, but not everyone does so properly, and that should be a point of concern for users.

Alex

[goMac]

The problem is an author could still write a bad APE module that has the same effect as it does in mach_inject. All APE does is kill application enhancing for a program if it crashes with application enhancing off. So sure, there is some protection, but in the end the plugin will still fail. Unsanity likes to certify plug ins as they go through of course, but an OS update could still cause trouble, and ultimately no one wants to go through certification and would rather use mach_inject. The reason Carpe Stellarem decided to write our own was because we a) were not interested in paying licensing fees, b) we doubted Unsanity would certify us because we are competition, and c) mach_inject lacked early load injection which we needed. If mach_inject had early load injection (with a daemon obviously) we would have chosen it.

The APE framework is also very very large. I would rather not have to install that with my application. The whole XTender Application Enhancer is about 30k. Its not yet as feature rich as APE, but it does the job perfectly well, just like mach_inject.

The problem is APE IS its large centralization. Programmers would rather work alone and not have to work through Unsanity. An APE module could go bad just as easily as a mach_inject based program. This is very evident by seeing that APE has yet to breach the commercial software market, while you have StuffIt using mach_inject. APE's approach is very Microsoftian, we'll check everything for you and allow you to install it on our framework. If the underlying framework ever has a bug, this could cause huge problems. With mach_inject, in addition to having the source code, you also have enough control to make compatibility changes on your own time.

I also find it interesting that many are complaining about Extension conflicts on Mac OS 9. I have been programming since System 6, and been using Macs since System 5 (512ke baby!) and I hardly ever had Extension conflicts (or System INIT conflicts for that matter). I often had applications crashing because they were poorly coded or old, but hardly ever extension conflicts.

If I ever did have an Extensions conflict, it was all because of load order. To go back to my example above, what happens if the beep override is loaded before the minimize override? Your beep override won't run because the pointer to it is not in the message chain.

System 6 and earlier had very little conflicts because Extensions did not exist. Everything a programmer ever needed to call was part of the operating system itself. Apple used something called System INIT's to add functionality. This was usually not functionality for developers, but functionality for the end user. These were pretty rare because everything you needed was in the OS anyway. You notice this is why System 6 boxes don't have icons running along the bottom at startup usually.

System 7 then came along and Apple decided over time that it was time to begin moving OS toolbox functionality out of the core OS and into modules. This is the time the "modern" System Folder was created. The Extensions folder was added, along with control panels (before this point System INIT's were just dumped in the System Folder, along with everything else). Over time everything became an Extension. Sound, Appearance (System 8), SCSI, USB, Firewire. System 8 was really the big one for moving everything to Extensions. At this point a lot of programs broke because those old core OS commands that they were looking for were disappearing in some cases. This wasn't the fault of an Extension conflict, this was the fault of Apple for moving everything. At this point, when writing extensions, it became necessary to make sure any Extensions you needed to use were already loaded for your Extension. For example, if you had an Extension which made use of USB, you had to make sure the USB extension was loaded first.

Developers on Mac OS/System 7 had some odd ways of getting around this. Extensions loaded in alphabetical order so you often saw Extensions with strange punctuation marks in front of the name or spaces.

There are still some signs of this era on OS X. Palm Desktop is perhaps the biggest. Palm Desktop has a daemon that runs in the background and looks on USB for when you hit the HotSync button on your Palm. This was not created as an Extension on Mac OS 9 because the USB extension would have had to have been loaded first. If the USB Extension wasn't loaded, Palm Desktop either wouldn't work or it would crash. So instead Palm created a startup item (now called Login Items under Mac OS X) to watch instead, which they knew would load after the Finder.

Not many people know this, but the fundamental idea of Extensions still exists under Mac OS X. They are now called Startup Items (not to be confused with the Mac OS 9 startup items mentioned earlier). Apple got wise and tackled the problems of Extensions on Mac OS 9. Startup items on Mac OS X have a priority system. For example, Networking may have a priority of first, while Apache may have a priority of last. this makes sure networking is loaded before Apache, or else Apache wouldn't have a network connection to start serving over. Apache also tells OS X that Networking MUST be loaded before it can load. You can specify what other startup items must be loaded first.

In short, Mac OS 9 extensions hardly ever had overlap problems. Most of the issues came from Apple moving functionality (to where sometimes an Extension couldn't find it to override and crashed) or load order. Extension conflicts are something that people seem to like to use as a reason why OS 9 is much better than OS X. And its true, OS X is better than OS 9, and Extension Conflicts did sometimes happen, but they were rare. The only reason I ever thought twice about installing an Extension was based on the age of the Extension and the age of my OS.

[CharlesS]

Originally posted by Basilisk:
Absolutely, patently false.

As a simple example, every single Cocoa application loads InputManager bundles. This is precisely how PithHelmet uses SIMBL to load itself in Safari. By default Cocoa applications have no direct control over this load, and AFAIK can't prevent it even if they want to.

You may believe that APEs are a different beast, but claiming that apps control all the plugins they load is false, and spreading tha claim precisely the kind of misinformation you lamented in your own post.

Straw man argument. I was talking about legitimate plug-ins for apps such as QuickTime codecs, not InputManagers.

This is what Apple's documentation says InputManagers are used for:

"The Cocoa text input management system centers around three classes that interact with each other to transmit input from the user�s keyboard or mouse to a text view. An input server (NSInputServer or other NSInputServiceProvider object) receives keyboard characters and transmits characters to be inserted into a text view object. An input manager (NSInputManager object) serves as a proxy for a particular input server and passes messages to the active input server. Finally, a text view (NSTextView or other NSTextInput object) displays, stores, and manages onscreen text."

That's odd, I don't see mucking with the behavior of windows, menus, and whatnot in that description. I wonder why that is? It's because people are abusing the input manager system to do things it's not meant to do. In fact, if you run class-dump on one of these InputManger plugins, you'll often find that there's not even a subclass of NSInputManager or NSInputServer in there at all. There's just a +load method designed to stick its tendrils into the Cocoa runtime, since the InputManager plugins get loaded early on in the process of launching a Cocoa app and NSInputManager apparently doesn't check to make sure one of those plugins actually is an input server or input manager before loading it.

If this is an endorsed method by Apple of enhancing applications, I haven't been able to find any documentation saying so. As far as I can tell, it's for adding input managers to the text input system, hence the name InputManagers. Claiming that it proves injecting code is a good idea is like claiming that the recent help: exploit is okay because you can do the same thing with the fantasy protocol attack.

No, it doesn't. I understand that this is a simplification that folks are using, but its not true except in the grossest terms and its been repeated so often that people have forgotten the real explanation of this.

Memory protection isn't "broken" by APE, it never existed in the absolute sense that folks seem to believe. A system that didn't allow processes to share memory would be useless. They could never communicate. You couldn't interact with them because they would have no way to receive events from the hardware.

There are documented, well-defined ways to pass or share memory between processes in MacOS X. One example is mach ports, and the various structures built on them (NSConnection, Distributed Objects, etc.). These are a feature of the OS and they are used pervasively throughout the system.

All too often in the recent rash of APE discussions people claim that these mechanisms are for debuggers only, but that's simply untrue. Everytime you change a hardware setting, everytime an keystroke event from your keyboard leaves the kernel, mach ports made that happen.

APE uses that same mach port mechanism to load code into other applications. You can argue that using that mechanism to patch code is potentially risky and I would agree (so long as you said "potentially"). However, memory protection wasn't violated any more than the kernel passing keyboard events violates memory protection. Both required different processes to share memory.

Okay. It is normal, desired, and endorsed behavior to pass around notifications, messages, Distributed Objects, etc. Applications are generally designed to receive such inter-process communication and handle them appropriately. It is not normal, desired, and endorsed behavior to patch code and screw with apps in ways that the author never could have anticipated or intended. The fact that you have to hack in using such means as pretending to be an input manager or a menu extra, or by using the interfaces intended for debuggers to use (which Rosyna said herself in the article), demonstrates this.

Not except in the broadest terms, both patch code. MenuCracker loads into SystemUIServer by pretending to be an Apple menu extra (a plugin that SystemUIServer allows to load). Once there it patches the selector for the Apple security mechanism to prevent it from blocking other extras.

Yes, again tricking the system into allowing you to do unsupported things.

Bottom line: If you're going to do dangerous and unsupported things, fine. But be honest about what you're doing. Don't try to mislead your customers into thinking that this is safe or that it uses supported mechanisms (abusing those mechanisms does not count as normal use).

[goMac]

Not to mention there are Apple endorsed ways of getting menu extras to run instead of cracking systemuiserver...

[CharlesS]

Originally posted by goMac:
The problem is APE IS its large centralization. Programmers would rather work alone and not have to work through Unsanity. An APE module could go bad just as easily as a mach_inject based program. This is very evident by seeing that APE has yet to breach the commercial software market, while you have StuffIt using mach_inject. APE's approach is very Microsoftian, we'll check everything for you and allow you to install it on our framework. If the underlying framework ever has a bug, this could cause huge problems. With mach_inject, in addition to having the source code, you also have enough control to make compatibility changes on your own time.

At least with APE, users know what they're getting, and know the risk. If something's doing its own mach_inject thing and the user doesn't realize it, this really does destroy memory protection since you'll have to reverse-engineer every app you run to make sure it's not patching something (witness QuicKeys and the other apps that are using mach_inject without saying so in the documentation).

I also find it interesting that many are complaining about Extension conflicts on Mac OS 9. I have been programming since System 6, and been using Macs since System 5 (512ke baby!) and I hardly ever had Extension conflicts (or System INIT conflicts for that matter). I often had applications crashing because they were poorly coded or old, but hardly ever extension conflicts.

If I ever did have an Extensions conflict, it was all because of load order. To go back to my example above, what happens if the beep override is loaded before the minimize override? Your beep override won't run because the pointer to it is not in the message chain.

System 6 and earlier had very little conflicts because Extensions did not exist. Everything a programmer ever needed to call was part of the operating system itself. Apple used something called System INIT's to add functionality. This was usually not functionality for developers, but functionality for the end user. These were pretty rare because everything you needed was in the OS anyway. You notice this is why System 6 boxes don't have icons running along the bottom at startup usually.

System 7 then came along and Apple decided over time that it was time to begin moving OS toolbox functionality out of the core OS and into modules. This is the time the "modern" System Folder was created. The Extensions folder was added, along with control panels (before this point System INIT's were just dumped in the System Folder, along with everything else). Over time everything became an Extension. Sound, Appearance (System 8), SCSI, USB, Firewire. System 8 was really the big one for moving everything to Extensions. At this point a lot of programs broke because those old core OS commands that they were looking for were disappearing in some cases. This wasn't the fault of an Extension conflict, this was the fault of Apple for moving everything. At this point, when writing extensions, it became necessary to make sure any Extensions you needed to use were already loaded for your Extension. For example, if you had an Extension which made use of USB, you had to make sure the USB extension was loaded first.

Are you serious? I had tons of extension conflicts on System 6 through OS 9. Yes, I'm including System 6. Extensions existed then - they were just called INITs. There wasn't that big a change in System 7 and above except Apple renamed the things to extensions. Yes, they started spinning off functionality by putting it in extensions instead of the core OS, but the nature of extensions was still basically the same. In System 6 or 7, didn't matter, I was still routinely having to rename extensions to get them to load in an order that wouldn't cause either a startup crash or other odd behavior. And often there'd be two that just wouldn't play nice with each other, period, no matter what order they loaded in. Thanks to my father knowing a lot of people in the small town where I grew up, I ended up being the Mac tech support for lots of his friends, family, colleagues, etc. during my high school years. About 99% of the really weird problems people had were caused by extension conflicts and were resolved like this: Start with shift key down, see if problem goes away. If so, remove half the extensions. If problem goes away, try the other half. If problem persists, remove half the rest. Repeat until you narrow the problem down to one extension. Now, try running all extensions except that one and see if the problem goes away. If so, try running with only that extension and the rest off. If the problem persists, it's a bad extension. If not, it's conflicting with something; repeat the entire process with that extension in the whole time to narrow it down to one other extension that it's conflicting with.

I used to repeatedly enter these instructions (or something like them) in response to lots of questions on Apple's Tech Exchange back in the day. As a reward, I got sent a free copy of OS 9 and a free Swiss Army Knife with an Apple logo on it, and got offered a T-shirt at some point as well. Hell, I even fixed a few misbehaving Macs at my school this way a few times. Thank God for Conflict Catcher, which was a godsend if someone had it. It would automate the entire process for you, so you wouldn't have to do all that crap by hand. So in summary, please don't try to tell me that extension problems were rare. I've been there. Extension issues were about as rare as casualties are during a war.

Not many people know this, but the fundamental idea of Extensions still exists under Mac OS X. They are now called Startup Items (not to be confused with the Mac OS 9 startup items mentioned earlier). Apple got wise and tackled the problems of Extensions on Mac OS 9. Startup items on Mac OS X have a priority system. For example, Networking may have a priority of first, while Apache may have a priority of last. this makes sure networking is loaded before Apache, or else Apache wouldn't have a network connection to start serving over. Apache also tells OS X that Networking MUST be loaded before it can load. You can specify what other startup items must be loaded first.

Except that those startup items don't patch traps. Big difference there.

[Basilisk]

Straw man argument. I was talking about legitimate plug-ins for apps such as QuickTime codecs, not InputManagers.

Begging the question. You asserted that plugins only load into apps when they request it. This is false. I wasn't aware that you had arbitrarily defined "legitimate plugins" as only those plugins which fit the requirement that they be loaded only on request. Any other tautological definitions you want to assert so that we can agree on common language?

If this is an endorsed method by Apple of enhancing applications, I haven't been able to find any documentation saying so.

Straw man of your own. I never claimed this was a Apple-sanctioned use of the mechanism. Just that applications don't get to choose whether to load them or not. Developers do not have absolute control over who they "invite into their houses".

The fact that you have to hack in using such means as pretending to be an input manager or a menu extra, or by using the interfaces intended for debuggers to use (which Rosyna said herself in the article), demonstrates this.

Straw man #2. Again, I have never claimed that this is not risky. In fact, if you review my posting history or my own software you will see I take great pains to explain that it is potentially risky. I believe I have a history of posting balanced information on this subject, see for example http://forums.macnn.com/showthread.p...hreadid=154961

What I do object to is blanket assertions that the mechanisms used are evil or breaking memory protection. These are untrue. If people want to educate about the risks, I'm all for that, but it needs to be factually correct.

Bottom line: If you're going to do dangerous and unsupported things, fine. But be honest about what you're doing. Don't try to mislead your customers into thinking that this is safe or that it uses supported mechanisms (abusing those mechanisms does not count as normal use).

I'll assume you've never seen my documentation. I have several paragraphs in the MenuMeters ReadMe that explain precisely why MenuCracker is necessary, along with cautions. Unsanity has a page of similar information, and these forums are full of cautionary tales that discuss the advantages and disadvantages without hyperbole.

Yes, developers have a responsibility to inform their users of risks. I assume that you do the same regarding Pacfist's use of Apple's undocumented package format (not just pax, which is of course, documented) and the risks of attempting to build a boot CD using totally unsupported mechanisms. Please do not assume that the rest of us don't take the same responsibility for our own code.

Alex

[Basilisk]

Not to mention there are Apple endorsed ways of getting menu extras to run instead of cracking systemuiserver...

No, there are ways to get NSStatusItems to run in the menubar. This is not the same thing as a NSMenuExtra. NSStatusItems have disadvantages, and I discuss the risks and tradeoffs openly and honestly with users in my ReadMe.

Alex

[Basilisk]

The problem is APE IS its large centralization. Programmers would rather work alone and not have to work through Unsanity. An APE module could go bad just as easily as a mach_inject based program. This is very evident by seeing that APE has yet to breach the commercial software market, while you have StuffIt using mach_inject. APE's approach is very Microsoftian, we'll check everything for you and allow you to install it on our framework. If the underlying framework ever has a bug, this could cause huge problems. With mach_inject, in addition to having the source code, you also have enough control to make compatibility changes on your own time.

I don't see this as a substnatial advantage, but I can see your concern as a Unsanity competitor.

One solution would be to have someone develop an APE alternative with an open source framework. It seems like this would be the best of both worlds. Eases competition concerns while still offering a centralized mechanism.

As for commerical software not using APE, I suspect that's just good old "Not Invented Here" syndrome. Recall that Unsanity actually did do Stuffit's MenuExtra for Stuffit 7 (and maybe 8 as well). Its not like they aren't speaking or anything

Not many people know this, but the fundamental idea of Extensions still exists under Mac OS X. They are now called Startup Items (not to be confused with the Mac OS 9 startup items mentioned earlier). Apple got wise and tackled the problems of Extensions on Mac OS 9. Startup items on Mac OS X have a priority system.

SystemStarter isn't really analogous to Extensions, its generally used to start control daemons. As Charles notes, rarely do they patch traps.

A better analogy is the IOKit probing system, which is undoubtedly smarter than MacOS 9 extensions.

In short, Mac OS 9 extensions hardly ever had overlap problems. Most of the issues came from Apple moving functionality (to where sometimes an Extension couldn't find it to override and crashed) or load order.

No offense, but I think you've redefined "extension conflict".

When users here talk about extension conflicts they almost always mean problems caused by load order. With the exception of breakages caused by actual moving functionality (with System upgrades) most conflicts were the direct result of stacking patches and overlap issues. If we drop that from the defintion, then sure, I never had a machine with an extension conflict either. But I don't think we could get most OS 9 users to agree with that definition

Yes users could adjust load order, and yes tools like Conflict Catcher could make this easier. But the fundamental problem was still overlap except in cases where older extensions failed on newer OS's.

BTW, you never did address my "surgeon's knife" question. Is it safe to assume that Duality injects into all processes? How then is this different than APE?

Alex

[CharlesS]

Originally posted by Basilisk:
Begging the question. You asserted that plugins only load into apps when they request it. This is false. I wasn't aware that you had arbitrarily defined "legitimate plugins" as only those plugins which fit the requirement that they be loaded only on request. Any other tautological definitions you want to assert so that we can agree on common language?

Not false at all. The Cocoa Input Manager, which is part of the Cocoa runtime, requests that Input Manager plugins be loaded into it. Cocoa is designed to accept those plugins, although an exploit exists to do things which the plug-in is not designed to do.

It's also completely irrelevant, since I was talking about application plug-ins such as QuickTime codecs, which was the example used by the person I was replying to.

The reason InputManagers are used to enhance applications is because someone found an exploit in them. This in no way makes any plugin like a QuickTime codec the same thing as an APE!

Straw man of your own. I never claimed this was a Apple-sanctioned use of the mechanism. Just that applications don't get to choose whether to load them or not. Developers do not have absolute control over who they "invite into their houses".

You implied it, by using InputManagers as a counter-argument to my statement that QuickTime codecs are not the same principle as APE's. And the fact that one plug-in mechanism can be exploited to do the things that an APE does is not sufficient to equate the two.

Straw man #2. Again, I have never claimed that this is not risky. In fact, if you review my posting history or my own software you will see I take great pains to explain that it is potentially risky. I believe I have a history of posting balanced information on this subject, see for example http://forums.macnn.com/showthread.p...hreadid=154961

I'm not talking about you specifically, I'm talking about the attitude presented by the article linked to in this thread (which is what we're discussing in the thread, right?). I don't know enough about your apps to say one way or another, but that article tries to claim that APE is safe and that it is using documented, sanctioned methods, since GDB uses the same hooks. It doesn't bother to mention that it is in no way recommended to run apps inside GDB for daily use.

Yes, developers have a responsibility to inform their users of risks. I assume that you do the same regarding Pacfist's use of Apple's undocumented package format (not just pax, which is of course, documented)

Undocumented, eh? Open PackageMaker, go to the Help menu, and choose Package Format Notes and see if you still think so.

and the risks of attempting to build a boot CD using totally unsupported mechanisms. Please do not assume that the rest of us don't take the same responsibility for our own code.

You're right about this one, which is one of the reasons that BootCD is sort of my "red-headed stepchild" compared to Pacifist, which I put more work into. It started out as a simple script-like app to automate the instructions for making a bootable CD which at the time could be found many places on the Net. I had no idea at the time that I would be expected to keep updating it for new OS releases, to figure out how to make various things work with it, etc. And it has problems - slow boot time, hardware and software configs that pop up all the time which cause it not to work properly, etc. I hate working with undocumented mechanisms, and there's a "What to do if your CD will not boot" file included with BootCD explaining the preferred way to send e-mail describing problems. Only a few days after releasing version 0.6, I've already discovered a few configs that don't play nice with it. It's a bitch.

At least if BootCD fails, it just creates a CD that doesn't boot, as opposed to APEs which can cause other apps to crash or behave strangely, causing angry users to write e-mails and VersionTracker reviews blaming the author of the app even though he has nothing to do with the actual problem. I hate that.

Oh, and before you bring up DockDisks, it sucks. That project has languished for so long because there's no real way to get it to do what I originally wanted it to. The original app was written in an afternoon as a "Let's see if I can do this" sort of thing (actually, BootCD started in a very similar way - there's a really old thread on here which contains the discussion which led to it) and has barely changed since. I'm going to relegate it to unsupported status sometime as the Finder sidebar does a better job of the same thing anyway.

[Basilisk]

Not false at all. The Cocoa Input Manager, which is part of the Cocoa runtime, requests that Input Manager plugins be loaded into it. Cocoa is designed to accept those plugins, although an exploit exists to do things which the plug-in is not designed to do.

Again, you've made an arbitrary distinction that suits your argument. The Cocoa text manager is designed to load plugins. mach_ports are designed to allow interprocess memory to occur. Both have Apple-sanctioned uses and unsanctioned uses. I agree that the unsanctioned uses may consitute abuse, but your response was to the following statement:

APE doesn't really break memory protection any more than plug-ins (such as QuickTime codecs or CMMs) do.

Notice that the discussion revolves around memory protection. You responded:

No, this is just one of these old wives' tales that manages to circulate amongst the general populace

And further:

Memory protection means that apps don't screw with each other. Apps can still support plug-ins if they intentionally do so. Adding a plug-in is like me inviting you into my house. A haxie is like you breaking my window and entering without permission.

This implies that plugins cannot "break" memory protection. Its not true, there are sanctioned uses of those mechanisms (like legitimate Text Input Managers), and by your own admission the sanctioned uses don't "break" memory protection even though they are automatically loaded.

Again, my argument is not that the unsanctioned uses aren't potentially dangerous, but claiming that these mechanisms break memory protection is misleading to users who only know "memory protection" as a vague concept.

Remember that what most users take away from these discussions is just a gloss of the technical truth. Its a disservice to them to use loaded terminology. Explain that patching is a bad thing all you want. Tell them you don't support your products with APE. Just don't imply that automatic code loading is "breaking memory protection".

It may just be me, but I feel strongly about keeping users informed on this stuff. The "wives tales" syndrome is hard to overcome. I just don't want any more tales to get started.

Its also frustrating to me that everything but APE is getting a free ride this week. goMac argues that mach_inject is a "surgeon's knife" but its pretty clear that Duality is poking itself into every app. And before the advent of inject based themers like ShapeShifter and Duality (the current version) the standard theming mechanism was to replace system files *shudder*.

If the conversation focuses on APE then users come away underinformed. They believe APE is bad, and ignore that their other favorite application does the same thing, just in a less visible way (I know this is a concern for you too based on your reply to goMac).

I'm not talking about you specifically, I'm talking about the attitude presented by the article linked to in this thread (which is what we're discussing in the thread, right?).

I think the discussion has gone beyond the contents of that post. I'll grant you that his use of GDB as an example was particularly unfortunate. The thing is, we're already shown in this discussion that there are lots of other ways to do this that don't involve GDB.

Undocumented, eh? Open PackageMaker, go to the Help menu, and choose Package Format Notes and see if you still think so.

I've seen it, I had assumed that you had gone beyond that content, espcially given that Pacifist broke on Panther. I assumed that was due to format changes. Was that because of other documented changes, or was that because you had gone beyond the documentation? Please understand I mean that in a complimentary way, I had always assumed that you had figured out a lot more than Apple reveals.

I hate working with undocumented mechanisms

I think we all do... I've just spent 2 days in GDB figuring our some obscure piece of the instant message framework just so I can organize my buddies into groups. At the end of the day though, we are sometimes left with little choice. Warn the users and move on has been my attitude, and given the popularity of the workarounds (MenuMeters, APE, Duality, BootCD, etc.) its pretty clear that on the whole its a win for the user.

Alex

[goMac]

Startup Item's are analogous to Extensions in the way that they add extra functionality to the system. Extensions were not really ever meant to do patching. If you look back farther in the history of the Mac OS they always added more functionality, and never modified the existing functionality. Around Mac OS 8 Apple began to use them for patching because the Mac OS was getting long in the tooth. Instead of modifying the core system they just began patching their own code. Before that point Extensions were only meant to add functionality. Extensions doing patching was a case of programmers being bad because Extensions were loaded into the System's memory space.

Startup Items are the same idea, except they are obviously not loaded into the System's memory space, meaning you can't override. Of course, you can go deeper to kext's to get into the system's memory space. Much functionality from Extensions is in startup items.

Here is everything the system handles via startup items on my machine:

AMD, Accounting, Apache, AppServices, AppleShare, AuthServer, BIND,
ConfigServer, CoreGraphics, CrashReporter, Cron, DirectoryServices, Disks,
IPServices, KernelEventAgent, LDAP, LoginWindow, NFS, NIS, NetInfo, Network,
NetworkExtensions, NetworkTime, Portmap, Postfix, PrintingServices,
RemoteDesktopAgent, SNMP, SecurityServer, SystemLog, SystemTuning, WebObjects,
mDNSResponder

A lot of those are Extensions in Mac OS 9. The only difference is they run in their own memory space. Overriding via Extensions was never something the original Mac OS was designed to do. Apple did a lot of things based on trust (remember no resources below 128?) and expected everyone to follow them. This is why shared memory failed.

As to the "surgeons knife" comment... Duality is meant to run in ALL Applications on the computer, therefore it is built with a daemon. However, if you have something like the earlier mentioned CVS Finder integration, you certainly don't want it in every single application. It may not hurt to have it run in all applications, but it certainly increases the risk of something going wrong.

I was comparing mach_inject and APE directly. I already noted the reasons we chose not to go with mach_inject, even though it was very heavily considered and we built a few models on it.

I work in tech support at a school district, so here is were the most problems happen with Extensions:

1) Teacher installs old program which comes with old Extension. Extension is incompatible, thrown away, teacher told to get new program, problem solved.
2) Teacher updates to newer Mac OS. Extension is incompatible. See 1. (This happened a lot when we moved everyone to Mac OS 9.2. It is Apple removing older code that breaks the Extension)
3) Hardware in the machine fails. Corresponding Extension fails and crashes machine. This has happened with modems and Firewire controllers sometimes.

The above reasons aren't conflicts as you can see. They are either old software or hardware failure. It was very rare for two extensions to play bad with each other.

Mac OS X has reduced the number of problems we're having, but there is still a significant amount. I'd say we get the same number of issues, but protected memory is probably keeping the end results of these issues from being as serious.

[CharlesS]

Originally posted by Basilisk:
Again, you've made an arbitrary distinction that suits your argument. The Cocoa text manager is designed to load plugins. mach_ports are designed to allow interprocess memory to occur. Both have Apple-sanctioned uses and unsanctioned uses. I agree that the unsanctioned uses may consitute abuse, but your response was to the following statement:

You know damn well that IPC is a much different thing than arbitrarily patching code, and that the former does not assume the latter. I don't want to waste the time to explain it all, since you are a programmer and you must understand what IPC is.

This implies that plugins cannot "break" memory protection. Its not true, there are sanctioned uses of those mechanisms (like legitimate Text Input Managers), and by your own admission the sanctioned uses don't "break" memory protection even though they are automatically loaded.

Straw man. I didn't say that plugins cannot break memory protection. Yes, no system is perfect and there will be exploits such as in InputManagers that get outside the limits that the plug-in is designed to have. Sure, I could invite you into my apartment, and you could slip something of mine into your pocket when I wasn't looking. The thing is that APEs are designed to do stuff they're not supposed to. Plugins are designed to have a very specific set of things they can do, and the program author has designed his app to be able to handle the plugins. APEs are designed to do stuff the author hasn't prepared for behind his back, without being invited in. They are designed for unsanctioned use; that is all that they do. To say that APEs do not violate memory protection any more than QuickTime codecs is like saying that you breaking my window and climbing in is no more illegal than entering my house when I invited you in. Period. Yes, certain specific examples of plug-in architectures can be exploited to do unsanctioned things. It's irrelevant.

Again, my argument is not that the unsanctioned uses aren't potentially dangerous, but claiming that these mechanisms break memory protection is misleading to users who only know "memory protection" as a vague concept.

Remember that what most users take away from these discussions is just a gloss of the technical truth. Its a disservice to them to use loaded terminology. Explain that patching is a bad thing all you want. Tell them you don't support your products with APE. Just don't imply that automatic code loading is "breaking memory protection".

But it is.

The designers of OS X wrote the whole thing specifically to prevent things like this.

It may just be me, but I feel strongly about keeping users informed on this stuff. The "wives tales" syndrome is hard to overcome. I just don't want any more tales to get started.

Then quit spreading them.

Its also frustrating to me that everything but APE is getting a free ride this week. goMac argues that mach_inject is a "surgeon's knife" but its pretty clear that Duality is poking itself into every app. And before the advent of inject based themers like ShapeShifter and Duality (the current version) the standard theming mechanism was to replace system files *shudder*.

Those things use the same exploit, and are just as bad. Maybe worse, since it's not as obvious that they're doing it.

I think the discussion has gone beyond the contents of that post. I'll grant you that his use of GDB as an example was particularly unfortunate. The thing is, we're already shown in this discussion that there are lots of other ways to do this that don't involve GDB.

But all involve doing something which is equally unsavory a hack.

I've seen it, I had assumed that you had gone beyond that content, espcially given that Pacifist broke on Panther. I assumed that was due to format changes. Was that because of other documented changes, or was that because you had gone beyond the documentation? Please understand I mean that in a complimentary way, I had always assumed that you had figured out a lot more than Apple reveals.

I had a mistake in the code, because I misunderstood something in the pitifully inadequate documentation for a certain framework. It didn't cause a problem under Jaguar because certain features advertised by the framework's documentation weren't implemented yet. Once a certain feature was implemented, it tried to kick in, and because I had a mistake in my code, it didn't work and threw errors all over the place. It was fixable easily enough once I figured out what was going on.

Since it started spitting errors right at program launch, it stands to reason that it didn't have anything to do with reading from a package format. Although, I see where you assumed that, since the package format did change completely between Puma and Jaguar. But that change was completely documented, and only required me to update the program to handle the new format.

I think we all do... I've just spent 2 days in GDB figuring our some obscure piece of the instant message framework just so I can organize my buddies into groups. At the end of the day though, we are sometimes left with little choice. Warn the users and move on has been my attitude, and given the popularity of the workarounds (MenuMeters, APE, Duality, BootCD, etc.) its pretty clear that on the whole its a win for the user.

I don't get as upset about using undocumented methods, since it won't mess up other apps or cause strange or erratic behavior outside the app itself. It has the problem of making your app break once the next OS version comes out, though. MenuCracker is not good, but I don't mind it as much as other haxies since it seems to only patch the menu extra check, and not other apps (such as mine). The whole idea of patching apps really makes me nervous, though. We can all think of worst-case scenarios - what if a haxie caused a path in an application to somehow get changed? What if that path got passed to -[NSFileManager removeFileAtPath:handler:]? What if the path used to be a specific file in my home folder, but after the haxie it got changed to the path to my home folder somehow? Far-fetched? Sure. But it could theoretically happen. Anything could happen when you're patching code that you don't completely understand, and which could change with any version update.

Oh, and goMac, there have always been extensions that patched things to give kewl behavior like animated cursors or whatnot, even back in System 6/7. Some were useful, some were crap, all had potential problems. I've experienced conflicts between these things, and had to fix them, going all the way back to 6.0.8.