Nothing inherently wrong with it, in the same way there's nothing inherently wrong with a relationship based entirely on casual sex with no attendant intimacy or commitment.
In both cases, though, in practice, you're likely to find one side is getting exactly what they want out of the relationship, where the other is holding out for more.
Staying friends with someone who you're romantically attracted to is a recipe for resentment (at least for many people). It might just be better to cut contact all together. Of course, then you might be in the uncomfortable position that people assume that you were only interested in her because you wanted to have sex with her and, 'failing' at that, you didn't want to have anything to do with her.
Ironically, it is when men aren't one-dimensional (which I think is must of us) that leads to these kind of situations, which leads to people thinking that they were just out after sex to begin with.
(This is a snarky way of saying that friendship is its own reward.)