Well, if you're "with" a person for some period of time, go places with him/her and act like a couple, then there is a strong and valid assumption on everyone's part that your having sex.
I've had a few guys in my life that have been absolutely wonderful friends and companions that I've gone places with etc, but have not had sex with because it would ruin a relationship ( believe me, elevating a friend to sex partner can absolutely wreck a friendship ) or, in one case, because he was gay. It was harder to explain that we weren't having sex than accepting that sex partners will talk. ( my gay friend was offered handjobs and blowup on many occasions but said he wasn't into girls. Whenever I see him, I offer a bj, hoping that maybe he'll say yes).
You date someone - these days the odds are you're having sex. And people talk. Do I mind? I would if it was in the middle of a big Christmas party or something and a bf bragged,a out the great blow jobs I give. I know I talk among my girlfriend about some guys, and of course they talk about us. Sometimes that's how you meet new people.