> This is in production right now as far as I know and adds very little latency.

How much? As GP said, > 5ms is instant disqualification. And that is for your whole app, not just the effects pipeline.

GP is right, go use a DAW in a serious setting for a while to build up an intuition of the needs of these users.