Pretty much this... incentives matter. But if you’re a slave, what’s your incentive to work harder?

As for literature backing this up, Adam Smith in Wealth of Nations talks in detail about slavery in the American colonies.

He gives examples where slavery was only viable in the southern states where they grew tobacco. Maybe one of the reasons why the South were so opposed to the end of slavery vs the North where slavery was expensive