Don't lump me in with the "Ally gurus"....I'm still learnin' too...every time I'm on the water.
But I will chime in with my $.02 worth....
As for skiing, what little I've done with my boat, it pulls just like other bass boats. Prop it right and it'll pull great. But I like to wakeboard and no bass boat is gonna throw a nice wake in comparison to a mastercraft and all bassboat hulls are gonna get pulled around a little when somebody cuts out hard and digs in with a wakeboard or a slalom ski. Basically, it ain't a mastercraft but it will get the job done...just getcha a good ski pole and a low pitched 4 or 5 blade prop.
For tubing, the Allison lacks a good tow point. The transom tie downs are really too close to the center to use for a tow point and I wouldn't recommend pulling a tube from a ski pole.
One area where the Allison does shine over other bass boats for family outings, skiing, etc, is the seating. In an XB2003 you can seat 5 comfortably (or 4 if you have a center steer setup). The layout is what really sold me on Allisons before I ever set foot in one. The wind protection in the XB2003 is great and if you get one with side by side seating, both you and your partner are protected from the elements.
You said yourself that you fish more than ski.....and the Allison is great for that. I've had several other boats with traditional bass boat layouts and the Allison fishes great. It is a little narrow but unless you are a really big bubba it won't take long to adjust.
You may say now that you aren't interested in running 100, but believe me when I say that speed is an addiction. The nice thing about the Allies is that you can have a good all around conservative setup and still run 90+. Like was said above, they run great with 150s on 'em, but you'll want more. I think just about everybody that powers 'em with anything less than a 225 PM seems to upgrade to a new powerplant in just a few years (if not sooner). But with the 150, it'll still run by most 225 powered tritons and be VERY efficient.
But the best advice I can give ya...find somebody near you that has an A-boat and spend a day on the water in one. But be prepared, one ride and you'll be boat shopping.
