The argument for unsafe ABI not being that big of a deal is that ABI boundaries often reflect organizational boundaries as well.

E.g. the kernel wouldn't really benefit from a "safe ABI" because users calling into the kernel need to be considered malicious by default.