I think most people who are concerned are bothered by the non-consensual activity. The people most likely to make public complaints about marital infidelity only do so for their political opponents and and voted for him by something like 90%.