I'm a 20 year old bi girl who has only been with men. Because, legitimately all of the girls I've ever had a crush on or been secretly in love with were straight (this is what happens when you're into fems as a queer woman). I am more than qualified to answer this.

I am still friends with most of the people I've had a crush on, or the people I don't talk to anymore are because they haven't been around since high school. If it's painful to be around her, that's when you have to pull the plug. If you still see her as a friend who you secretly want to hold the hand of or f***, that's ok. But if every time you hangout with her, this stresses you out, or you can't help but think about, or you start to become the embodiment of /r/niceguys or /r/nicegirls, that's when you have to leave.

I hope this helps, OP.