[agentz]

Evening all
At work, supporting close on 100 seperate schools we have a whole pile of Airport base stations, ranging from original graphite ones through to the last Extremes before Apple killed them in Europe.

We're having a pile of grief with a fair number of them, specifically Extremes of various models.

The basic version of the problem is that they just freeze up, they stop transmitting wireless signals, the LAN port on them goes dead, but they can still be accessed by Airport Admin via the WAN port, so the network connectivity is obviously still there. A quick reboot either from Airport admin or by yanking the power and up they come. Sometimes they'll stay up for days or weeks on end, and at other times they'll only stay up for five minutes before locking up again. When it happens the WAN led on the front is still indicating traffic (unsurprisingly) but there's just no indication at all from the wireless indicator.

All of the ones we've been having issues with have been updated to firmware v5.7 on the recommendation of our dealer, and at first things seemed to settle down for a few weeks, but then things started acting screwy again.

One school in particular is driving us nuts. It's got five base stations, three Extremes, one snow dual-usb and one graphite. The snow and graphite ones have never once fallen off the network, but the Extremes are up and down like yo-yo's.

We've moved base stations around, we've swapped base stations out with ones that have been working perfectly in other sites. We've tried all sorts of channel combinations, interference robustness off and on, auto channel negotiation, restricting them to 802.11b only, .11g only, .11b/g compatible, tweaked the multicast rates, transmitter power, station densities, all just about every combination of configurable options on the base stations.

They're connected to a fully switched network with a local Win2k3 server dealing with DHCP and the like. We've turned off all access control and authentication, no difference.

We came across a known issue with certain Cisco switches and Extremes whereby they apparently can't auto-negotiate their WAN port speeds, so have set the switch ports and base station WAN ports to 100mbps/full duplex manually. No change.

I set up one of the more problematic base stations to write a debug log to a Mac running syslogd and I noticed some slightly odd behaviour, but it doesn't seem to correlate with periods when the base stations just disappear. Last Friday I was sitting watching what the hell was going on when I noticed the base station I was monitoring seemed to reboot. The syslog was sitting there logging clients associating and disassociating quite merrily when the next thing I see is 'Jan 01 2002, 00:00 - initializing firmware' and then the base station going through it's initializing process til it went off to the ntp server and synch'd its clock. It did this three times over an hour, but not once did it freeze up and vanish off the network for any more than 20 seconds or so.

My gut feeling is that there's something environmental screwing with the wireless stuff, but I've got no idea what the hell it could be.

I may have missed some of the troubleshooting we've gone through, but my head is mince from days of thinking about this stuff.

Has anyone got any suggestions about what the hell we can try next?

PS: I should say that we've got multiple sites with the same sort of configurations that have never had a problem.

[ghporter]

Heat and weird traffic are the AXBS's worst enemies (this is true for almost all routers, but especially so for AirPort routers). Funky traffic can cause the firmware to sieze up and require a reboot, and the "funk" factor is dependent on so many things that it's impossible to define the problem set. Regular rebooting (say midnight on Sunday) is a good practice, and you could probably craft some sort of script to tell all of the AEBSs you're managing to do this. The tiny little microcomputer that is the guts of an AEBS needs a little TLC sometimes because it was never designed for mission critical usage.

You should also make sure that they're positioned for best heat management. Closets are BAD for electronics that don't have their own fans, so if that's where you install these things, PUT A FAN IN! And probably an exhaust duct too.

Good luck!

[agentz]

Originally Posted by ghporter 

Heat and weird traffic are the AXBS's worst enemies (this is true for almost all routers, but especially so for AirPort routers). Funky traffic can cause the firmware to sieze up and require a reboot, and the "funk" factor is dependent on so many things that it's impossible to define the problem set. Regular rebooting (say midnight on Sunday) is a good practice, and you could probably craft some sort of script to tell all of the AEBSs you're managing to do this. The tiny little microcomputer that is the guts of an AEBS needs a little TLC sometimes because it was never designed for mission critical usage.

You should also make sure that they're positioned for best heat management. Closets are BAD for electronics that don't have their own fans, so if that's where you install these things, PUT A FAN IN! And probably an exhaust duct too.

Good luck!

Hi Glenn
Thanks for the feedback. I've been seeing a lot of comments online about weird traffic causing base stations to freeze up, but as you say it seems to be impossible to track down exactly what sort of traffic can cause this.

I've been nosing through the syslogs for one of the base stations I'm monitoring at the moment and I've been seeing some dot11isassociation messages with the reason being either 'received invalid class-2 frame' or 'received invalid class-3 frame'. Again however, I've not been able to spot any correlation with instances of this and the base stations freezing up.

There do seem to be a lot of associate/deassociate messages in the syslogs, which to me looks like client machines going into State 2 (authenticated but unassociated), and occasionally they'll drop to State 1 (unauthenticated and unassociated), for no apparent reason, but I don't know enough about 802.1x stuff to go much further than that.

I think I forgot to mention in my original post that the clients in the site I'm having grief with are a mixture of iBook G4 with AEX cards and Dell Latitude D600/D610's with either Dell Truemobile or Broadcom wireless cards. I believe the AEX cards and the base stations are based on a Broadcom chipset?

I did see some mention of Broadcom cards having the MAC sublayer of Layer 2 in software rather than firmware, so somebody elsewhere was suggesting that there may be a client or two on the network with some sort of corruption of the MAC sublayer that's causing weird State1/2 behaviour and confusing the hell out of the base stations, but I don't know how likely (if at all) that could be.

As for the location of the base stations, they're all mounted on a walls out of the reach of kiddie fingers, so heat mangement should not be an issue.

As a slight aside re: rebooting the AEBS, I was at one of my other sites that is having the same problem when one of the class teachers collared me.

She says to me "we had a parents evening last night and one of the parents who works in IT says that the problem I'm having with this base station is because I switch it off each night. He says they don't have any memory so when I switch them off they lose all their settings and that's why they don't work".

Um, no. Ignore what the 'parent who works in IT' says and carry on doing what you are doing and powering down the base station at the end of each day - if anything that should be helping the problem you're currently experiencing. Gah, dontcha just love 'people who work in IT'?

[ghporter]

"People who work in IT" can be word processing specialists... And as you know just about everything nowadays has some flash memory so turning things off is NOT a problem.

I'm not sure about the chipset in AEX cards, but goofy client activity certainly can goof up any access point-at least any "consumer level" access point. I know that your school system went with AirPort because it was less expensive to implement, but with any large infrastructure you really should have commercial-grade equipment, such as Symbol or Cisco access points, because they're built to handle the type of traffic this sort of environment generates, and more importantly, their operating software is built closer to "mission critical" standards, so they work better for longer between goof ups. My university has a bunch of Cisco access points up on the walls all over the place, and they just plain run. A friend works in network security here, and he has pointed out more than once that he'd much rather "watch" the wireless network do its thing, consistently and reliably, than corral non-compliant grant winners who think they own the school and can do whatever they please with the network, no matter what it does to thousands of other users..