There is a tendency to conflate the two because they are strongly correlated. It's not a mistake to assume that Tyrone is poor and black and Chip is rich and white -- it's a best guess based on observations about reality in the US.

Does it make it absolutely moral to make that assumption? Of course not, people should be treated without regard to previous biases, etc. But it doesn't mean the statistical inference is invalid.