Every character on the show does similar, bad things so it's always confusing to me when people single out one character. I think it's difficult to perceive what Damon did to Caroline when she never even calls it rape. Long before Elena and Damon slept together, Caroline and Damon were friendly. She'd say every now and then that she doesn't like Damon, but she then chooses to help him and they hang out and laugh and work together.

Caroline goes through a similar experience with her feelings for Klaus, which other characters including Elena bring up many times in season four. Despite everything he did to her, her boyfriend and her friends, she has feelings for and she chooses to sleep with Klaus. Stefan was raped by Katherine, but he never says it and in fact, he and Katherine sleep together before she died which was very messed up if you think about it. But then Stefan raped lots of women too which Elena sort of knew about in season three when she went to Chicago and Damon even read her lines he wrote out of a diary. So Elena and Caroline being with Stefan has the same sort of weight. But because that happened to women they will never know, it's easier to desensitize and not worry about it.

So it's just a giant, hypocritical mess when you think about it.