[Peter]

Our OS X Server keeps randomly kernel panicking, and leaving very vague crash/kernel logs...

Crash Log:

Unresolved kernel trap(cpu 1): 0x700 - Program DAR=0x00000000E0476000 PC=0x0000000000020004
Latest crash info for cpu 1:
Exception state (sv=0x2F738C80)
PC=0x00020004; MSR=0x00089030; DAR=0xE0476000; DSISR=0x42000000; LR=0x0001FF8C; R1=0x181F3CD0; XCP=0x0000001C (0x700 - Program)
Backtrace:
0x0001FF8C 0x000242C8 0x00049834 0x0002921C 0x000233F8 0x000ABAAC
0xFFD5D5D5
Proceeding back via exception chain:
Exception state (sv=0x2F738C80)
previously dumped as "Latest" state. skipping...
Exception state (sv=0x2F2CC000)
PC=0x9000AB48; MSR=0x0000D030; DAR=0xE0476000; DSISR=0x42000000; LR=0x9000AA9C; R1=0xF01806D0; XCP=0x00000030 (0xC00 - System call)

Kernel version:
Darwin Kernel Version 8.8.0: Fri Sep 8 17:18:57 PDT 2006; root:xnu-792.12.6.obj~1/RELEASE_PPC
panic(cpu 1 caller 0xFFFF0007): 0x700 - Program
Latest stack backtrace for cpu 1:
Backtrace:
0x00095138 0x00095650 0x00026898 0x000A7E04 0x000AB780
Proceeding back via exception chain:
Exception state (sv=0x2F738C80)
PC=0x00020004; MSR=0x00089030; DAR=0xE0476000; DSISR=0x42000000; LR=0x0001FF8C; R1=0x181F3CD0; XCP=0x0000001C (0x700 - Program)
Backtrace:
0x0001FF8C 0x000242C8 0x00049834 0x0002921C 0x000233F8 0x000ABAAC
0xFFD5D5D5
Exception state (sv=0x2F2CC000)
PC=0x9000AB48; MSR=0x0000D030; DAR=0xE0476000; DSISR=0x42000000; LR=0x9000AA9C; R1=0xF01806D0; XCP=0x00000030 (0xC00 - System call)

Kernel version:
Darwin Kernel Version 8.8.0: Fri Sep 8 17:18:57 PDT 2006; root:xnu-792.12.6.obj~1/RELEASE_PPC
Model: PowerMac3,5, BootROM 4.3.3f2, 2 processors, PowerPC G4 (2.1), 1 GHz, 1 GB
Graphics: ATI Radeon 7500, ATY,RV200, AGP, 32 MB
Memory Module: DIMM0/J21, 512 MB, SDRAM, PC133-333
Memory Module: DIMM1/J22, 512 MB, SDRAM, PC133-333
Network Service: Built-in Ethernet, Ethernet, en0
PCI Card: ADPT,1686806-04, scsi, SLOT-4
PCI Card: PDC20269, ata, SLOT-5
PCI Card: pci10ec,8169, sppci_ethernet, SLOT-2
Parallel ATA Device: WDC WD2500JB-00GVA0, 232.89 GB
Parallel ATA Device: HL-DT-ST CD-RW GCE-8240B,
USB Device: Smart-UPS 1500 FW:601.3.I USB FW:1.5, American Power Conversion, Up to 1.5 Mb/sec, 500 mA
USB Device: Hub in Apple Pro Keyboard, Mitsumi Electric, Up to 12 Mb/sec, 500 mA
USB Device: Apple Optical USB Mouse, Fujitsu Takamisawa Component, Up to 1.5 Mb/sec, 100 mA
USB Device: Apple Pro Keyboard, Mitsumi Electric, Up to 12 Mb/sec, 250 mA
FireWire Device: Macintosh, Apple Computer, Inc., Up to 400 Mb/sec

Any clues as to what would cause this? Its really annoying having our server randomly crash..!

[Sherman Homan]

That is vague, does the log only ever reference CPU 1? Try this, I know it is not optimum, but it might isolate the problem, disconnect the USB to the Smart-UPS, remove the software. Leave the UPS in place doing its thing, just kill the USB connection and Smart stuff. I have had many problems with it.

[Peter]

another crash log...

panic(cpu 0 caller 0x000A4AC8): simple lock (0x003D8020) deadlock detection, pc=0x000333CC

Latest stack backtrace for cpu 0:
Backtrace:
0x00095138 0x00095650 0x00026898 0x000A4AC8 0x000333CC 0x00031710 0x000313CC 0x0003D91C
0x000B3E54 0x000AD7D4 0x000ABD48
Proceeding back via exception chain:
Exception state (sv=0x2D3CBC80)
PC=0x907CEA54; MSR=0x0200D030; DAR=0xF0284EC4; DSISR=0x42000000; LR=0x907CEA20; R1=0xF0C26FA0; XCP=0x00000024 (0x900 - Decrementer

[proton]

The first log backtrace resolves to:

Code:
Creating stack map... loading modules and nametables... done.
Unresolved kernel trap(cpu 1): 0x700 - Program DAR=0x00000000E0476000 PC=0x0000000000020004:
        Backtrace 1:
                0x0001ff8c: _ipc_right_destroy
                0x000242c8: _mach_port_destroy
                0x00049834: _mach_port_server_routine
                0x0002921c: _ipc_kobject_server
                0x000233f8: _mach_msg_overwrite_trap
                0x000abaac: _ppcscret
                0xffd5d5d5: iNullCall
        Backtrace 2:
                0x00095138: _print_backtrace
                0x00095650: _Debugger
                0x00026898: _panic
                0x000a7e04: _trap
                0x000ab780: _thandler
        Backtrace 3:
                0x0001ff8c: _ipc_right_destroy
                0x000242c8: _mach_port_destroy
                0x00049834: _mach_port_server_routine
                0x0002921c: _ipc_kobject_server
                0x000233f8: _mach_msg_overwrite_trap
                0x000abaac: _ppcscret
                0xffd5d5d5: iNullCall

That backtrace suggests to me that the backtrace has been corrupted. This often occurs due to a locking bug in the kernel or a kernel extension. There is very little you can do in this situation unless you're an experienced developer who isn't afraid of some two machine kernel debugging.

The second log isn't a complete log so I can't be sure the backtrace I extracted from it is correct. The panic message does indicate the kernel detected a locking issue (a deadlock in this case), the backtrace indicates:

Code:
Creating stack map... loading modules and nametables... done.
panic(cpu 0 caller 0x000A4AC8): simple lock (0x003D8020) deadlock detection, pc=0x000333CC:
        Backtrace 1:
                0x00095138: _print_backtrace
                0x00095650: _Debugger
                0x00026898: _panic
                0x000a4ac8: _slckPatch_isync
                0x000333cc: _thread_setrun
                0x00031710: _thread_go
                0x000313cc: _thread_timer_expire
                0x0003d91c: _timer_call_shutdown
                0x000b3e54: _rtclock_intr
                0x000ad7d4: _interrupt
                0x000abd48: _ihandler
        Backtrace 2:
                0x00095138: _print_backtrace
                0x00095650: _Debugger
                0x00026898: _panic
                0x000a4ac8: _slckPatch_isync
                0x000333cc: _thread_setrun
                0x00031710: _thread_go
                0x000313cc: _thread_timer_expire
                0x0003d91c: _timer_call_shutdown
                0x000b3e54: _rtclock_intr
                0x000ad7d4: _interrupt
                0x000abd48: _ihandler

That's about all the information I can provide from those panic logs at this stage. Can you provide any further information about what software is installed on the server that might help diagnose the issue?

- proton