NTA by a long shot.

First of all, the women in your neighbourhood have no right to tell someone that YOU would drive them wherever they want. That's just not right. If they are so concerned about your neighbour, I'm sure they can drive her themselves or ask their significant others to do it.

Secondly, I feel that your neighbour is also being very unreasonable. I personally would never ask my neighbour, no matter how well I know them, to take a day off work and drive me somewhere which is 4 hours away. That's literally insane.

And about "would you want your sister to be treated that way."- your sister is your family, someone you've known your whole life. The two comparisons are absolutely stupid.

Plus, you handled it perfectly well. You conveyed to your neighbour that you had no idea about what the others had said. If your neighbour is a reasonable human, she will see that you are nowhere in the wrong.