Texture memory (TMEM) is very special and fast, so it tends to be expensive stuff, at least back then, and it's balanced together with the rest of the architecture, like the bandwidth the RDP has, DMA copies from main mem -> TMEM, and so on. It would have changed quite a lot of the underlying architecture to increase it to 8KB, and might not have been worth it. You couldn't have increased texture resolution by simply increasing memory without changing a significant part of the architecture elsewhere (e.g. RDP fill rates). Most textures used during draws on the N64 don't even fill up the 4KB texture memory.
That being said, going to a real cache rather than a block of manually managed memory I thin would have been the better design choice. Most textures didn't fill up the memory because of the practicalities of managing that memory. A double buffer scheme to load a block wile rendering from another one, the fact that you have to eat the whole cost of the full texture's load before you can render from it, etc.
