I have also been thinking on this after last session, as well as the sword-in-one-hand, pistol-in-the-other-hand issue. On AoO I have decided the rule will stand for the following reasons. Pistols of the time period were a little different from what we have experienced firing modern handguns.
1. The pistol has to be held level and the arm fully extended for accuracy and to prevent it flying up into the face of the user upon discharge.
2. A pistol isn’t all that small. Average length for a standard pistol was probably close to 2 feet. Almost standard crossbow sized.
3. You have to plant your feet firmly and stay still while firing. The action on a flintlock pistol takes the better part of; or maybe more than a second to complete. That in my opinion, is the main reason there are AoO for ranged shots at all (you can’t defend yourself or move out of the way).
4. There exists a feat which negates the penalty so it gives you something to strive to.
5. The rule applies to your enemies as well as players. That is how you were able to destroy Captain Sosas blunderbuss before he could turn Herrick into a blood sausage.
Perhaps I could wave the AoO for pocket pistols.
As for sword-in-one-hand, pistol-in-the-other-hand issue. I think taking Two-weapon Fighting feat should negate the penalty for firing the pistol in the off hand completely as Two-Weapon Fighting is a combined feat that used to include Ambidexterity. If you were to use the pistol as a melee weapon as a second attack to your sword, you would still incur the -2 to both attacks. Make sense?