[RevEvs]

Im thinking of a small project which involves monitoring a FTP server, and AFP, SMB on OS X.

How would I be able to go about accessing the FTP server? i.e. so my app can detect/be notified when someone logs in, or when they change directory or access/create/delete a file.

I know the hard way would be to run tcpflow and monitor all the data and then do a lot of parsing, but am wondering if there is a better way?


revs.

[Ludovic Hirlimann]

Originally posted by RevEvs:

I know the hard way would be to run tcpflow and monitor all the data and then do a lot of parsing, but am wondering if there is a better way?

Well you could configure your ftp server to log everything done and then daily perform an analysis of that log (but this is not real time).

[utidjian]

Originally posted by RevEvs:
Im thinking of a small project which involves monitoring a FTP server, and AFP, SMB on OS X.

How would I be able to go about accessing the FTP server? i.e. so my app can detect/be notified when someone logs in, or when they change directory or access/create/delete a file.

I know the hard way would be to run tcpflow and monitor all the data and then do a lot of parsing, but am wondering if there is a better way?


revs.

There is a pretty simple command for watching logs called "tail". Simply configure your ftp server to log everything then run:

tail -f /var/log/ftplog

(or whatever your ftp log file is called). By default tail will show you the last 10 lines (adjustable) of the log file. As users log in and out and do stuff... if it is logged you will see it.

If you are using the wu-ftpd ftp server then you can run commands like ftpwho, ftpcount, and xferstats. To have them update in (almost) real time run:

watch -n 1 ftpwho

or something like that.

For some reason this question seems to be asked pretty often... but I am never quite sure why someone would want to monitor ftp users that closely.

[Ludovic Hirlimann]

Originally posted by utidjian:

For some reason this question seems to be asked pretty often... but I am never quite sure why someone would want to monitor ftp users that closely.

Warez ??

[utidjian]

Originally posted by Ludovic Hirlimann:
Warez ??

So why would this require monitoring your ftp users in real time? If one or more of your users is doing "illegal" transfers... simply lock them out.... end of problem.

Unless you have a very slow site and have a lot of time on your hands to sit watching ftp transfers... you are not neccessarily going to see a "warez" transfer when it happens. It is much simpler and more practical to use the powers of grep and your log files to spot these transfers than to sit there watching the monitor.

How are you going to determine what is and is not a "warez" file? "my_carib_vacation.jpg" could be a "warez" file that was renamed... "my_carib_vacation.zip" could be a whole collection of them. Are you going to open every single file AS they come in?

If you are running an anonymous ftp server the usual (and default) way to allow uploads is that they go into /wherever/incoming where incoming has write ONLY permissions. Anon. users can not see nor download what is in there. The admin can then, at their convenience and discretion, distribute the files from incoming to /wherever/pub.

All I am saying is... ftp has been around for a loooong time. Aside from the obvious security problems of non-anonymous ftp (just don't use it) anonymous ftp and how to use it has been pretty well worked out. It makes about as much sense as opening and reading all the contents of all the briefcases as people enter an airport.

[RevEvs]

Im only asking this as I will shortly be writing a program, and part of it will be just doing some FTP monitoring (and other protocals), hence the realtime updates that i need - as i'll want the application to be up to date - not just being updated every now and then.

Just bringing this back on topic before it turns into a warez/ftp/anonftp thing

Im messing around with a networking program, and was just wanting to see if i had ways to hook into the various servers in the default mac os x install (my application would be a lot easier to do if i just included my own ftp/ssh etc servers - but i thought i should try and make it work with the default install - people dont want new stuff being installed everywhere!)

So back on topic, any other unix programs,scripts (preferably in the default install) give informatiion on the connected users for ssh/ftp/afp.. any help appreciated - I dont want to have to parse everything from the TCP/IP data flow if i dont have to.

cheers

revs

[utidjian]

Originally posted by RevEvs:
Im only asking this as I will shortly be writing a program, and part of it will be just doing some FTP monitoring (and other protocals), hence the realtime updates that i need - as i'll want the application to be up to date - not just being updated every now and then.

Just bringing this back on topic before it turns into a warez/ftp/anonftp thing

Im messing around with a networking program, and was just wanting to see if i had ways to hook into the various servers in the default mac os x install (my application would be a lot easier to do if i just included my own ftp/ssh etc servers - but i thought i should try and make it work with the default install - people dont want new stuff being installed everywhere!)

So back on topic, any other unix programs,scripts (preferably in the default install) give informatiion on the connected users for ssh/ftp/afp.. any help appreciated - I dont want to have to parse everything from the TCP/IP data flow if i dont have to.

cheers

revs

OK... I will leave the "warez/ftp/anonftp thing" alone for now.

I think I already mentioned some utils for monitoring FTP traffic. There are many others.

I looked at tcpflow and I don't really think that is the right tool (or component) for what it is you are trying to do.
Speaking of which... are you trying to make something like a p2p client/server package... that shows all sorts of stuff regarding what the p2p server is up to? If so that is something that is built on to the particular p2p software. To do something similar would require rebuilding all the services you want to watch or creating modules for each service that talk to your monitoring program. Part of that functionality is already there in each server program in that they use the syslog facility. As I already mentioned you can monitor syslog for the interesting stuff. Another one is netstat (see the manpage for that). I don't know what Mac OS X uses as an interface to the kernel... Linux uses the /proc filesystem. Using whatever the equivalent of /proc is in Mac OS X you could build your program to access that info directly and massage it into a palatable form.

Incidentally... if you can pick up the August 2003 issue of Linux Journal the "Cooking with Linux" column deals with this topic directly. I won't go into any in depth descriptions but here is a list of resources/programs you can look up:

netstat
Nnetstat www.aneptun.de/linux/Nnetstat
contrack-viewer.pl cv.intellos.net
Driftnet www.ex-parrot.com/~chris/driftnet
pkstat http://www.itee.uq.edu.au/~leonard/p...tware/#pktstat
IPTraf iptraf.seul.org

There may be native Mac OS X versions available or fink ports of the programs. In any case... some interesting stuff.

[RevEvs]

what i want to do is monitor the default networking servers in OS X, then provide a nice GUI.
I know if i could rebuild, or bundle my own servers it would be easy, but as im trying to make it work with a default OS X install. Im just trying to find any way of getting information about the running network services.
FTP seems easiest as i can intercept it easily with tcpflow, and get plain text with the FTP codes.
for AFP i know i could use the log, which would at least let me know when someone logs in or out, or creates/deletes a file/folder.
And Ive been looking at this <i>configd</i> to see what info i can get from there.

Hopefully this will by a Uni project, not sure yet - just doing some research to find out if its feasible.

cheers for the info.

revs

[utidjian]

Originally posted by RevEvs:
what i want to do is monitor the default networking servers in OS X, then provide a nice GUI.
I know if i could rebuild, or bundle my own servers it would be easy, but as im trying to make it work with a default OS X install. Im just trying to find any way of getting information about the running network services.
FTP seems easiest as i can intercept it easily with tcpflow, and get plain text with the FTP codes.
for AFP i know i could use the log, which would at least let me know when someone logs in or out, or creates/deletes a file/folder.
And Ive been looking at this <i>configd</i> to see what info i can get from there.

Hopefully this will by a Uni project, not sure yet - just doing some research to find out if its feasible.

cheers for the info.

revs

OK well tcpflow is the wrong tool I think.... if I am understanding correctly. tcpflow is for capturing a copy of the files/traffic as it goes by. Unless you have a specific purpose to do exactly that it doesn't make much sense to capture the entire traffic content. Again it is as if you are opening every briefcase that passes through an airport and making a photocopy of the contents. If that is what you want to do then, yes tcpflow will do that. If you want to open every briefcase, note the title of the contents, determin how long the briefcase is in the airport, its origin, destination, who is carrying it, etc... and do the same for every backpack and suitcase and pocketbook... then netstat and some of those other tools I mentioned are useful for that. The logs also do this for you... it is really their reason for existence. There are some very good tools for analyzing them... such as webalizer (http://www.webalizer.com) which can be used to analyze ftp, web, mail, and other things.

For afp I am not sure. Are you sure you can really see/log what files are created and deleted via afp?

There are also some ethical issues in how much logging and monitoring is done on a server. The makers of sendmail specifically did NOT include the facility to create extensive logs and mirroring of emails other than sender, recipient, and size of the email. They feel that anything more than that makes it too easy to invade someone elses privacy. Most fileservers are also built with that kind of philosophy in mind.

Another problem with attempting to watch all transfers in real-time is that some of them are so quick (most?) that you won't seen anything useful... humans just cant read that fast. That is why logfiles were created. On even a msall departmental server the level of activity can be so great that it is not possible for a single human to read the logs generated in a normal workday. That is why we use grep and monitoring tools to look for the interesting stuff for us. Things like webalizer (there are others) for condensing all the data in the logs to a useful, human readable format. How are you going to monitor a connection to and ftp server that lasts for less than a second or two?

It was Doug Gwynn who said, "UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things."

[RevEvs]

As for AFP, i can montior the log file to find when a user logs on and off and if they create/delete files/folders.

As for monitoring all the data - My applciation will have things such as "Mr Anon has signd in to FTP" and such, im sure ill be ignoring a load of information, ill be only picking up thing such as users logging in and logging out, and users modyfying the filesystem by creating files or something.

As for ethical issues, yup i agree, and as this hopefully could be a Uni project then the discussion of ethical issues will probably be good in the report

Im just toying with the idea of making the monitoring of filesharing and networking on OS X more user friendly, I know people perfectly capable of running the FTP server, but wouldnt have a clue about logs.
Also, i was fed up of always paying attention to logs or network spikes when my friends would log on to my computer and mess around when i want the bandwidth
Anyway, if i go ahead with this, i have some cool ideas that should benefit some people, if not just me, and whatever happens ill improve my programming/cocoa skills

thank you for all the help so far, greatly appreciated.

revs