Yes, you're right. Average human reaction time is on the order of hundreds of milliseconds though. If you can keep latency under ten milliseconds you're probably doing pretty good, and that's reasonable over distances of a few hundred kilometers. Having the operator thousands of kilometers away is only really necessary for extrajudicial assassinations, which should be put to a stop anyways.

I can remember playing Quake on a server with a latency of 40 ms and being an untouchable god because of it.