I've always felt like I need 3 hands to use it. It is like 2 controllers welded together. In more complex games I have to constantly change my grip from the outside left to the middle position and I've just never gotten used to it.
...there are games which use the analogue stick and the D-pad? They shouldn't do that --- you're supposed to hold it either by the two outside wings, with d-pad and buttons; or by the middle and the right-hand wing, with analogue stick and buttons. Using the analogue stick and the d-pad makes no sense, for exactly the reasons you say.
Okay, let me rephrase: the N64 controller has designed for people with at most two hands. Use the d-pad with the left hand and the analogue stick and trigger with the right hand is fine. Requiring the use of the buttons as well is not.