That's right, in fact if people remember that secret then it's not a "second factor" it's just another part of their password. A "factor" in the context of authentication means one of the various ways that can be used to verify someone's identity: "something you know" (password), "something you have" (non-duplicatable object, eg a SIM card or OTP token containing a secret that cannot be easily guessed or extracted), or "something you are" (biometrics).
> in fact if people remember that secret then it's not a "second factor" it's just another part of their password.
This is more generous than it should be. Your TOTP secret is just another part of your password regardless of whether you personally remember it or not; what matters is that, if I would like to be you, I only need to know the secret.