You either can be outsourced inexpensively, or you can't. If you can, but your physical presence in a Bay Area office is effective in stopping your employer from figuring it out, then you should probably stay here.

If being able to work for any company remotely means I have to compete with the whole world, I’m ok with it.

Would you be willing to compete wage wise? It's completely possible to find programmers that speak English for single digit hourly wages.

I wouldn't be competing with the single digit programmers, but yes. Geography still matters for remote employees (due to time zones and the need for an occasional face time) as does the level of English and obviously the level of experience. The pool of people that fit that description isn't so large that I would have to work for minimum wage from the comfort of my mountain cabin.