[tobster]

Hi all,

I'm getting constant kernel panics on a new Mac Pro here at the office. Seemingly at random the dreaded gray multi language screen flows down the display.

The computer is a Mac Pro 2.66 ghz stock machine, with 3GB RAM. It came like this from Apple and no other modifications have been done.

I'm at a loss as to how to find what is causing this. I'm not entirely sure when this started but it certainly hasn't been going on from the beginning. The computer is running a mixture of Photoshop CS2, Photoshop CS3 beta and Capture One. In my experience this is happening most when CS2 is idly open and/or in use. CS2 is also prone to crashing on the machine at random. Another identical Mac Pro in the office is working just fine doing the same type of work.

Any help is greatly appreciated.

Thx, Tobbi

[Big Mac]

Please post a Panic Log (found in /Applications/Utilities/Console)

[Sherman Homan]

Kernel panics can run the gamut from simple permissions errors to hardware problems. And sometimes combinations of several things. The log can (sometimes!) help narrow it down.

[tobster]

Hi,

Thanks for the quick replies. Here is what I found for today's date under panic.log in Console:

*********

Mon Apr 9 15:52:00 2007
panic(cpu 2 caller 0x001D078B): vnode_put(71617bc): iocount < 1
Backtrace, Format - Frame : Return Address (4 potential args on stack)
0x378a3998 : 0x128d1f (0x3c9540 0x378a39bc 0x131df4 0x0)
0x378a39d8 : 0x1d078b (0x3d1c4c 0x71617bc 0x71617bc 0x6960e70)
0x378a3a08 : 0x1d0841 (0x71617bc 0x7513e3c 0x0 0x0)
0x378a3a28 : 0x1be0c8 (0x71617bc 0x6820800 0x0 0x0)
0x378a3f68 : 0x378337 (0x64f51f4 0x6960e70 0x6960eb4 0x0)
0x378a3fc8 : 0x19acae (0x681aea0 0x0 0x19d0b5 0x67cd2dc) No mapping exists for frame pointer
Backtrace terminated-invalid frame pointer 0xb019db28

Kernel version:
Darwin Kernel Version 8.8.1: Mon Sep 25 19:42:00 PDT 2006; root:xnu-792.13.8.obj~1/RELEASE_I386


*********

Mon Apr 9 16:11:23 2007
panic(cpu 0 caller 0x001D078B): vnode_put(718b000): iocount < 1
Backtrace, Format - Frame : Return Address (4 potential args on stack)
0x377f3998 : 0x128d1f (0x3c9540 0x377f39bc 0x131df4 0x0)
0x377f39d8 : 0x1d078b (0x3d1c4c 0x718b000 0x718b000 0x6949a98)
0x377f3a08 : 0x1d0841 (0x718b000 0x6a1013c 0x0 0x0)
0x377f3a28 : 0x1be0c8 (0x718b000 0x6818c00 0x0 0x0)
0x377f3f68 : 0x378337 (0x650c1f4 0x6949a98 0x6949adc 0x0)
0x377f3fc8 : 0x19acae (0x6843380 0x0 0x19d0b5 0x6843380) No mapping exists for frame pointer
Backtrace terminated-invalid frame pointer 0xb021f778

Kernel version:
Darwin Kernel Version 8.8.1: Mon Sep 25 19:42:00 PDT 2006; root:xnu-792.13.8.obj~1/RELEASE_I386


*********

There are older dates there but none of the log info means anything to me so I can't say if they are the same or not. All do have the "Backtrace terminated-invalid frame pointer ..." line at the end.

Thanks,

Tobbi

[Big Mac]

Unfortunately, that kind of panic is harder to diagnose. Sometimes there's a specifically isolated driver family in the log that let's you know it has something to do with a particular piece of hardware, but that's not the case here.

[tobster]

Thanks for that. I assume its best (and least time consuming) then to just reinstall system and application software? Rather than try and track down the real reason for this (assuming it is not hardware related)...

Today is a public holiday here in Iceland so it is not that much of a problem to do it tonight and have the machine up and running for the work week starting tomorrow

- tobbi

[olePigeon]

Find the CD/DVD that has your Apple Hardware Test (it'll say AHT on it in little letters, might be the same disc as your operating system; follow the directions on the CD/DVD) and run a quick check (shouldn't take long.) If that doesn't come up with anything, run the thorough check just before you leave your office or just before you leave for lunch. This'll rule out any major hardware poblems before you try and reinstall any software. It'd be a waste of time to reinstall everything if it was just a bad RAM stick.

[mfbernstein]

Run a RAM test overnight (memtest is the most complete tool for this). Also, if you have a full backup, you can try upgrading to 10.4.9 (use the combo updater though), and see how that helps.

[tobster]

Ok, hardware tests are over (normal + extended) and results say everything is working properly.

Next step, update to 10.4.9 (nothing to back up as there is only system + applications on the drive, all files are stored on Xserves).

Will report back.

Tobbi

[Aelfinn]

I'd open the box and make sure everything is seated properly--the memory and riser cards, the graphics card, all the drives, etc.--just to be sure.

[tobster]

So, all of the above suggestions done:
* box opened up and memory modules refitted
* hardware test DVD run - everything passed with flying colours
* system updated to 10.4.9
* permissions repaired

System seems to be running ok now, at least no panics yet (crosses fingers). Total re-install is in order if it starts acting up again.

Thanks everyone for your help!

- tobbi

[tobster]

So - system has been acting up again. The problem is, an iMac in the same office is doing the same. It seems to me this is always happening when browsing files on shared folders on a local Xserve running OS X Server. It might also be happening when accessing shared folders on a local G4 running OS X standard.

Following is the log from the iMac:

panic(cpu 0 caller 0x001D1888): vnode_put(3b6eb58): iocount < 1
Backtrace, Format - Frame : Return Address (4 potential args on stack)
0x1ca63988 : 0x128d08 (0x3cb134 0x1ca639ac 0x131de5 0x0)
0x1ca639c8 : 0x1d1888 (0x3d38a4 0x3b6eb58 0x1ca63a18 0x140867)
0x1ca639f8 : 0x1d193e (0x3b6eb58 0x4ea063c 0x0 0x0)
0x1ca63a18 : 0x1bf228 (0x3b6eb58 0x3a24c00 0x0 0x0)
0x1ca63f58 : 0x379e23 (0x38b9dac 0x3e11ac8 0x3e11b0c 0x0)
0x1ca63fc8 : 0x19b17e (0x39007e8 0x0 0x19e0b5 0x390165c) No mapping exists for frame pointer
Backtrace terminated-invalid frame pointer 0xb021eb28

Kernel version:
Darwin Kernel Version 8.9.1: Thu Feb 22 20:55:00 PST 2007; root:xnu-792.18.15~1/RELEASE_I386

What do these "frame pointer" errors mean?

Thx, Tobbi

[dimmer]

The 'best' reference I've seen to this (Frame Pointers) is:
Calling Convention

I suspect bad RAM, naughty RAM, but honestly we as consumers shouldn't be looking into things this deeply: try a reformat and reinstall if you feel like it, but if you can live without the system while it gets repaired / replaced I'd go that route.

[tobster]

Hi again,

So, now it seems all the intel Macs in-house are going kernel ballistic on me whenever I browse the Xserve shares. Very odd. The PowerPCs might do it as well, but as of yet I have been unable to recreate the problem there. Now what I can't figure out, is it a problem with the intel machines (2xMac Pro, 1xiMac) or is it a problem with the Xserve?

I read through the site dimmer linked to and I have to admit I don't understand much of what is written there.

I'm pretty much at a loss right here hoping that it is not the Xserve that is causing this. Signs point to it since it is happening via several clients.

Any ideas?

- tobbi

[kick52]

how do the computers connect to the Xserve?

[bearcatrp]

I would disconnect from the server, run your programs, to see if its the server causing it. If not, then you rule out the server causing problems.
Randy

[tobster]

2xPowerMacs + 1xMacPro + 1xiMac all connect through two gigabit switches. 1xMacPro connects through one gigabit switch. They connect to the G4 server in the same way.

- tobbi

[tobster]

Originally Posted by bearcatrp 

I would disconnect from the server, run your programs, to see if its the server causing it. If not, then you rule out the server causing problems.
Randy

Thats the problem, I seem to only be able to recreate these kernel panics when browsing that particular Xserve's shares.

- tobbi

[Sherman Homan]

Could be: a bad ethernet card in the Xserver. They are cheap, swap it out. I think the frame reference is referring to ethernet packets.
A bad switch, but it looks like you have two switches, do the Macs on each switch panic?

[tobster]

Yes, they panic on each switch. I even tried swapping out switches (I always have one backup, just in case) and it still happens.

The Xserve has two Ethernet connectors. Maybe swapping between the two is worth a try? If indeed one is broken I'll contact Apple. The Xserve was bought 65 days ago.

- tobbi

[Sherman Homan]

I would pop in a new ethernet card. I assume you have one card for the WAN and one (the suspect!) for your LAN?

[tobster]

I've been browsing through the logs on the Xserve trying to find anything that helps me from that end... I have no idea if the following has anything to do with this, but maybe someone here can shed some light.

From the system.log.0.gz:

Apr 25 03:15:04 vinnsla cp: error processing extended attributes: Operation not permitted
Apr 25 03:18:01 vinnsla servermgrd: servermgr_dns: no name available via DNS for 192.168.1.12
Apr 25 03:18:01 vinnsla servermgrd: servermgr_dns: no hostname set and unable to detect via DNS, services may not function properly - use changeip to repair

The IP 192.168.1.12 is the IP for the Xserve from where the log comes.

Hmmm...

- tobbi

[tobster]

Originally Posted by Sherman Homan 

I would pop in a new ethernet card. I assume you have one card for the WAN and one (the suspect!) for your LAN?

I only have one of the ethernet ports on the Xserve hooked up. That goes into the switch in the rack. None of the computers here are available outside our LAN.

Here is a photo of the network setup (note, HUBs are actually gigabit switches):



The circled computers are the ones having problems (Xserve included although the panics are not happening there).

I'll wait for everyone to stop working and try the other ethernet port. See if that helps. I'm only new to using an Xserve (learning as I go here ), but do I need to specifically tell the Xserve I'm swapping ports?

- tobbi

[Sherman Homan]

[edit] I type too slow...!
You are not using the other ethernet card?
Yes, you will need to tell the Xserver about the big change. Copy the settings for the card you are using.
TCP/IP, DNS zones, NAT port forwarding, etc.
Apply those to the other card, plug and play.
Yeah, you should wait until your workforce goes home...!

[bearcatrp]

On the macpro that keeps crashing, how are the ram modules aranged and how is the 3gb set up (2 1gb and 2 512mb)?
Randy

[proton]

Sigh

This is very unlikely to be to do with the network ports in the Xserve. The frames referred to in the panic log are stack frames, a perfectly normal part of any program (you couldn't implement functions very easily without them). They are not ethernet frames.

The panic log indicates the panic is occurring in vnode_put_locked, part of the VFS (virtual file system) code. In fact, the specific line of code that is triggering this panic is line 3214 of bsd/vfs/vfs_subr.c (Mac OS X 10.4.9 / xnu-792.18.15).

Ultimately, the call is from getattrlist, and judging by the proximity to the end of the function of that frame in the backtrace, is probably caused by line 1290 of bsd/vfs/vfs_attrlist.c calling vnode_put. This looks very suspiciously like a locking issue in the kernel, they tend to appear a lot more on machines with more cores (for relatively obvious reasons).

Do you have a lot of files in the directories on the Xserve? Can you reproduce the problem if you copy the files you're browsing to the local machine?

About your DNS error logs from the Xserve: Mac OS X Server needs correct DNS configured. This means fully working forward and reverse DNS must work for the server's name and IP address or odd things will happen. The changeip tool has a changeip -checkhostname option to help diagnose these problems.

- proton

[P]

If you want to try disabling one CPU to see if that helps, the command is

nvram boot-args=cpus=1

Run it as root, and then reboot.

[tobster]

Originally Posted by proton 

Sigh

Ultimately, the call is from getattrlist, and judging by the proximity to the end of the function of that frame in the backtrace, is probably caused by line 1290 of bsd/vfs/vfs_attrlist.c calling vnode_put. This looks very suspiciously like a locking issue in the kernel, they tend to appear a lot more on machines with more cores (for relatively obvious reasons).

Do you have a lot of files in the directories on the Xserve? Can you reproduce the problem if you copy the files you're browsing to the local machine.

- proton

Hi Proton,

Thanks for the answer and the very extensive and informative info. The problem goes away if I make a copy of the folder causing the problem at each time to either the local computer or simply duplicate it on the Xserve. The funny thing is, like yesterday morning, browsing/opening a folder called, say... "Panic Session" , would cause a kernel panic, then I would restart and it would be able to browse/open that folder fine. Shortly afterwards another different folder caused a panic.

The panics are seemingly random, yet I've only experienced them while browsing a network mounted drive or share. Until a few days ago the problem seemed to be happening when browsing the Xserve only but then while connecting to the studio Mac Pro the user Mac Pro panicked. The G5s in the office (all dual) are not reporting any problems doing the same. The intel iMac has had a few of these (though far fewer than the Mac Pros).

- tobbi