How much does size matter?

I've heard the adage there's no replacement for displacement, and in live events it certainly seems that with a wide open space you want the big guns and a ton of power.  "Sometimes you do need the Cadillac to tow the boat".  Bringing it back home, however, I have some fairly large speakers (DCM KX-12 Series Two - 12" woofers, bigger enclosures), and some smaller ones (Bose 401 - 6.5" woofers and a smaller volume enclosure).  I've been to houses with a pair of speakers similar in stature to the Bose 401's and been blown away by the performance.  You'd honestly think in a few cases with a powerful amp set to flat (no bass, no treble added) that you had a subwoofer stashed somewhere helping them out.  How do they do that?!

My wife was kind enough to point out early in our marriage that if I hadn't bought the DCM floorstanding towers before we were together, I wouldn't have them.  I can't imagine no other male has ever encountered such a statement. 

I realize that 401's aren't really small speakers (see Bose cubes, table top radios, and various satellite speakers, for example), but they are volume-wise probably a third of the DCM's.

In today's age (albeit what's under the hood of speakers is basically unchanged for some time), can one make a strong argument that you have to have large speakers to get the whole picture?  Let's assume for sake of argument that we aren't going for concert sound levels, but if we can get there, all the better.  A friend of mine always points out it's easier to turn it down than to turn it up.  A dealer customer of mine also once told me that the first watt is the most important (how often and for how long are we really moving a lot of power through our speaker wire, honestly?); doesn't mean that I don't want a bad-ass looking set of speakers if I have the means though!