Welcome to the MacNN Forums.

gooffer · Jul 16, 2005, 11:07 PM

Well very simple running big server moving lot's of files, once I hit about the 100000 file mark scp-ing files it crashes and that is the end of the xserve.

Everytime it happens I have to call someone and send them to the server to manually reboot.

Anyone know what's happening there? Or am I the only one pushing these amount of individual files onto an xserve.

Not very impressed by this xserve G5....

gooffer · Jul 16, 2005, 11:55 PM

I guess I should clearify, sshd seems to hang. Crashlog shows nuttin.

bmedina · Jul 17, 2005, 12:03 AM

If it's useless to you, you could send it my way....

Detrius · Jul 17, 2005, 12:32 AM

Originally Posted by gooffer

I guess I should clearify, sshd seems to hang. Crashlog shows nuttin.

Maybe using VNC would help diagnose... In the energy saver settings, there's a "Restart automatically if the computer 'freezes' " option. This works by setting the PMU to restart the machine in five minutes. Every few minutes, it resets the timer. If the timer ever doesn't get reset, the PMU automatically reboots the machine. I'm not exactly sure what the times are, but that's the idea. So, if the machine isn't automatically rebooting for you, then it's still running. You could also write a script for launchd that checks to see if sshd has hung and restarts it.

gooffer · Jul 17, 2005, 11:30 PM

Well re-phrase again, when you get to the actual server the machine does not appear to hang it is the sshd that is frozen and there is no way to relauch it without having it rebooted. My guess is that it is related to OS X running out of file descriptors but I am not sure if this makes sense. Something is making it run into trouble, is there some limitation on how many files can be transfered ?....

Might have to contact apple about this some time, I have a couple of more worms for them to look at. Coming to think of it is the swapping of harddrives on an Xserve and same problem that you get in windows were C becomes D or E or vise versa after a reboot and you can never reformat a drive before first checking which one is which, very anoying to say the least.

Jellytussle · Jul 18, 2005, 02:46 PM

I've not tried using scp with that number of files, how about looking at more appropriate tools, rsync for example? Could you be a little more descriptive of your issue with drives? If you mean the fact the entry in /dev is not consistent, i believe that's a consequence of the way OF sets up the device tree at boot. There are other ways to uniquely identify a volume.

gooffer · Jul 18, 2005, 11:07 PM

2 million files, adding sometimes a hundert thousand a day all new once. Rsync makes no sense, would be way to slow I know exactly which files are new. Hence scp is the only way to go. If there wasn't that weird time-out, hanging of sshd or a way to relaunch it I would be fine. I am still thinking file discriptors have to have something to go with it. Anyone know if there might be a set limit.

Jellytussle · Jul 19, 2005, 02:09 AM

have you looked at sshd when you're running your transfer? fstat may tell you something. Why would scp require that many open file descriptors? you could also bump up kern.maxfiles above 12288 with sysctl; but i think you might be better off examining your methodology. if you know your changed files exactly, then perhaps make up a BOM file and pipe them through ditto over ssh? rsync would also let you do this ( not with a BOM file), and has a very efficient client/server mode. What's the application here?