It has also been found that diets high in cholesterol and/or saturated fat also lowers testosterone [1,2,3]. I think the best course of action is to reduce rather than eliminate. i.e. "control" your intake of cholesterol and saturated fats. Blindly increasing your intake of either (above acceptable levels) goes against many scientific studies.

[1] http://www.drmirkin.com/men/1008.html [2] http://www.ergo-log.com/toomuchcholesterol.html [3] http://www.theatlantic.com/health/archive/2012/03/study-of-t...