Well... you have a few potential things to think about:
Typically the bigger switches have much better backplanes. That means that when things start to get busy they perform much better. More traffic and less collisions.
You are probably talking about "dumb switches" with the 8 port variety... very often these are just switches in the sense that each port can be either 10/100 and simplex/duplex... but they are effectively hubs when it comes to routing information. Once again, at even moderate traffic levels this has large performance benefits (advantage on the true switch).
You seem to only be talking about a single switch, so we can skip the whole border and routing protocols part of the conversation (small "switches" don't do it).
Lastly, the larger switches tend to have nice management functions. When something starts to go wrong you can find it and fix it. With the smaller devices this can be days worth of work.