Sex is supposed to be about intimacy, which is more than just close physical contact. It's about complete trust and acceptance of the other person, faults and all, and a willingness to be completely open and vulnerable to them in return.
So, as far as sexual fantasies go, I have heard it both ways. It all depends on whether or not the relationship is based on honesty or if it's based on falsehoods. Is the other person included in the fantasy? Are they aware of it and okay with it? If so, and as long as there is no physical danger then a sexual fantasy is probably not harmful and might even be beneficial.
On the other hand, if one person has a sexual fantasy, and they keep it a secret from their partner, then there is a good chance that the entire relationship is already in trouble because it lacks trust and intimacy. The person with the fantasy, especially if they indulge in it without telling the other person what they're doing, is not in a relationship with the other person at all - they are in a relationship with their own imagination and a faceless body which they are using for their own gratification. That's not love, and it's certainly not intimacy.
Some fantasies might be physically impossible, some might be illegal, some might be genuinely dangerous, so it might be a very bad idea to indulge in them, but in a good relationship it should still be possible to talk about them, and perhaps find a related compromise that appeals to both partners.
I don't know if this helped or if it made things even more confusing, but in any case -
Welcome to Apositive.