Don't feel sorry at all. As a dude who at one time couldn't take a hint if his life depended on it, I've been in this situation before. Gal wanted to have a date, but at the last minute said she was stuck at home. So being the people pleaser I try to be, I offered to bring the food to her, so we could still have the date. We sat around and watched a movie, but when I tried to get close, she actively backed away. In that moment I 1000% understood what the situation was, asked her if the whole situation was uncomfortable for her, and when she said yeah, I excused myself from her house.

Just saying no from the start is the best way to avoid heartache and headache on both parties involved, but you are ABSOLUTLEY in your right to cut things off at any point, and don't let anyone convince you otherwise. It's not rude to have personal boundaries and enforce them, regardless of how you feel.