It goes back to the late 1800's and Early 1900's and like many legal overreaches in the US, zoning laws were primarily spawned from a desire to discriminate on the basis of race or class.
I generally avoid citing Wikipedia directly but in this case I feel it gives a pretty digestible overview while being fairly well sourced throughout. Even if it's not perfectly Fair & Balanced^tm I don't think people in the early 1900s using the law to suppress minorities/the poor is a such contested viewpoint that we need to hunt for the perfectly centrist perspective to get a rough idea.
https://en.wikipedia.org/wiki/Zoning_in_the_United_States#Or...