Yeah, technically your phone isn't even involved. The watch manages all the cryptographic keys independently and you can add cards through the Google Wallet app or website.
It's not that bad, my battery lasts around 36 hours. I take it off when I go to shower and by the time I remember to put it back on (~30min) it's usually back to 90%+.