* It can be phished. An active phish persuades you this is Famous Bank, you enter the TOTP code, they relay it to the actual Famous Bank and steal your money.

* Under the hood it's just a Shared Secret which means the Relying Party can lose all your credentials. You have no way to be sure they did a good job securing them, and they've got every reason to blame you if those credentials are stolen.

* Psychologically humans perceive the TOTP code entry as mutually authenticating when it isn't. The site asked for my Famous Bank TOTP code, and it worked, so therefore this is actually Famous Bank. This actually makes them less cautious than they should be.