Aside from needing an obvious connection and desire for the person on the other end of the line, phone sex is really dependent on creativity and how skilled someone is with dirty talk and weaving a fantasy with another person.
In many ways, phone sex and cyber are similar to writing erotica. You are creating a scenario and verbally exploring a fantasy, except in this case you have a built-in audience of one that is giving you instant feedback. It's sort of like 'collaborating' with another person, except it's all spontaneous, in-the-moment improv. Phone sex can be mad seductive, dirty, intense and uninhibited. But... as I'm sure the story goes, as with writing erotic stories, some do it well, and others don't.
Your love or hate of phone sex is probably shaped by whatever experiences you've had with it. If you got a total 'dud' (like if all you've read in erotica was a story you'd like to vote with a score of 1), you're probably not going to become a fan of it or see how others can find it hot.