[blairmacintyre]

Hi all,

I figure this is a short in the dark, but it can't hurt to ask.

As part of a research project I'm running here at Tech, we wish to monitor everything a user is doing on their desktop (we do a variety of things with this information, such as creating visualizations of activities, background searches for relevant documents and email messages, etc.).

right now, we use Windows 2000 as the desktop, since we know how to do the monitoring on windows (in a fairly crude way).

I'd personally love to replace the desktop machine with a MacOSX machine (since I prefer the mac). But, I've never done a lick of Mac programming (done a lot of Unix programming for X and NeWS, and some Windows programming).

So, before heading there, I'd like to know if it is even possible to do what I want, which includes:
- notification when apps start and stop
- notification when focus changes
- notification of everything that is typed
- the ability to capture window shots of running applications while the user is working
- the ability to do "virtual window manager" type activities (show and hide windows, move them around, etc)
- the ability to find out what documents an application has open, where the user is in the document, etc.

Any comments, pointers, etc., would be appreciated. While I would obviously like as much info as possible, I'm more interested in knowing if people think it's possible or not, even if they aren't sure how ...

thanks.
blair

[Angus_D]

Some of this may be possible... There is a vnc server for OS X for getting screen and typed commands, I think. I'm not sure about everything, I don't think you'd be able to do all of it.

As for the documents, you'd have to make sure the application responds to commands you send to it. You could write your own custom program to respond, if you like.

[Lolo from Paris]

You mean something like Backorifice for Windows ???

With VNC you can control and see what the user does on his desktop but i don't know if you can have the informations and save them in a text file for example...

Timbuktu had more options but it's only OS 9 currently :-/

[Dalgo]

You mean you want Big Brother to watch over your macs? Hmmm... ever see the 1984 Apple ad? I don't think that Apple would put such nice 'hooks' into their OS.

[knobby]

Originally posted by Dalgo:
You mean you want Big Brother to watch over your macs? Hmmm... ever see the 1984 Apple ad? I don't think that Apple would put such nice 'hooks' into their OS.

Well, I suppose Big Brother would appreciate the help, but I if Apple included such hooks it'd be a nice way to statistically test the GUI.. For example, how far (average) does the user tend to move his/her mouse between clicks? How far do they miss button by, and how often? Which directories are the users accessing most frequently? etc..

[blairmacintyre]

Originally posted by Dalgo:
You mean you want Big Brother to watch over your macs? Hmmm... ever see the 1984 Apple ad? I don't think that Apple would put such nice 'hooks' into their OS.

Big brother has nothing to do with our, although obviously the things I need to do could be used for Big Brother activities. The system we've built uses a couple of extra machines (plus some projectors) to augment your current office environment. Info is not shared with anyone but yourself on your own machines.

Anyway, I don't want to get into that. The technology we need (being able to capture keyboard events, notice when windows receive focus, capture window images) are typical window system functions, and let you do things like get screen grabs (for printing or saving to disk), capture keyboard activity to monitor for global shortcuts, etc: there is no reason that they wouldn't be in OSX, but I wanted to sanity check before I started.

[Angus_D]

Why exactly do you need to do this? Sounds like the ultimate employee checkup tool for companies. I certainly wouldn't work for someone who used this to check up on me.

[blairmacintyre]

Originally posted by Angus_D:
Why exactly do you need to do this? Sounds like the ultimate employee checkup tool for companies. I certainly wouldn't work for someone who used this to check up on me.

The OS facilities we need could also be used for monitoring employees (no surprise there), and there have been programs written to do that (I recall reading about them at some point). That's not what we want them for, of course.

Since you ask, I'll try to give a quick summary.

One of my research projects (I'm a CS prof at Georgia Tech) involves looking at how we might use wall-sized displays in personal offices. By wall-sized displays, we currently mean using things like SmartBoards or LiveBoards (which are basically front or rear projected displays that have pens are and fairly large, about 3 or 4 feet across).

There are lots of uses for these things, especially in meeting rooms, classrooms, and so forth. Lots of researchers hypothesize that they will become a common part of offices in the future, and have shown how you might use them in a variety of ways (one of my collaborators has done work on this in the past).

One of the things we want to do, in addition to using them as electronic whiteboards, is to display historical information about what you were doing in the past on your desktop computer(s).

Have you ever used a multi-desktop virtual window manager (such as olvwm or tvtwm or CDE on unix, or Virtual on the mac)? What those things do is let you have mutliple desktops that you can flip between. An easy way to think about what we are trying to do is to visualize each of these multiple desktops on a part of the electronic whiteboards, so you can see what you were doing in all these tasks that are not visible on your desktop.

Most of the stuff I asked about is needed in window managers (to capture images of the windows for display, to move windows around, to know the layout of your windows, etc.). Other things, like capturing all keyboard events and knowing exactly what documents you are editing/accessing and where you are in them, are needed e want to provide some "advanced" features, such as:
- re-opening old documents that you had used a while ago, in the place you left them (as you can do from the "recent docs" list, but we want to allow distant historical access on a per task basis)
- automatic searching of databases (the web, your email, your files) for documents related to what you are currently doing (this is based on work folks in information retrieval do, typically called "rememberace agents")

So, the idea is to integrate your desktop/laptop computers with wall sized displays in your office to create a more effective computing environment. The stuff I'm asking about is one small part of this larger effort. We are well aware that there are huge privacy implications in this sort of work, which we are thinking about.

Anyway, I hope this helps ...

cheers,
blair

[knobby]

Blairmacintyre, your project sounds sounds pretty cool.. Good luck with it..

I have no idea how OS X handles mouse/keyboard events but there must be a way (it may be undocumented at this point) to slide something between the Darwin and Quartz layers. You amy want to take a look at XFree86 and see how they get the mouse position from Darwin, or FreeBSD, or Linux..

Good luck..

[blairmacintyre]

Originally posted by knobby:
Blairmacintyre, your project sounds sounds pretty cool.. Good luck with it..

I have no idea how OS X handles mouse/keyboard events but there must be a way (it may be undocumented at this point) to slide something between the Darwin and Quartz layers. You amy want to take a look at XFree86 and see how they get the mouse position from Darwin, or FreeBSD, or Linux..

Good luck..

Thanks ... I've done hacking inside an X server before, so I know how the X to Darwin layer will work. Usually, once the window system grabs the keyboard and mouse, the lowest level access won't work.

I'm sure there will be ways to get the information I want, I was hoping someone might have some pionters before I send a student (or myself) scurrying through all the documentation!