I think you're drawing a false parallel, and I hope you're not serious. An action which has a victim is not necessarily a crime, and an action which is deemed a crime does not necessarily have a victim. Note that this is a practical distinction, not an ideological one. In a pure ideological social-contract style system one could argue that the basic definition of a crime should be an action which infringes on the rights that we as a society agree to possess.

On a very basic level, illegalizing things because they 'negatively impact people' would bring capitalism to a grinding halt.