So I could build a mansion or a small house and pay the same, just because I'm paying for the zoning. And of course zoning could change and much of it can also have dual use. This complicates the calculation and can lead to surprises.

It seems current actual land usage through property tax better accounts for the way the land is being used, is more stable, less arbitrary, and implicitly complies with the existing zoning.

Without reforming zoning, it is largely useless. Sure it might cause some improvements in places underutilizing their land but I think most places are using their land for the highest value thing it is zoned for.

If I have a SFH zoned for SFH in the heart of manhattan next to skyscrapers, the value of the land isn't based on the skyscrapers, it is based on what you can put on it and you can prove that by selling it for which nobody is going to pay as if you could put a skyscraper there.

"underutilizing their land but I think most places are using their land for the highest value thing it is zoned for."

In urban or higher density suburban areas, I think you're mostly right. But in lower density areas, a lot of the land is multi purpose or has few restrictions. So technically, you could build apartments or homes on land that is used as agriculture. However, there might not be demand for it in that area. Essentially, the tax-by-zoning works well in the situations where zoning is restrictive and demand is high. But in low density areas where the actual use is not well defined with a single zoning use, the value would be potentially arbitrary or inaccurate (making property tax a better measure of value).