Provided someone can easily walk away, I would say not to stress about it. Different story if someone starts following you, physically gets in your way, etc. I'm friendly in a dumb dog kind of way, so I'm not bothered by people talking to me about random shit in most settings. A bus is kind of annoying because they'll ignore a book, etc, anyway.

When I worked in retail, one customer took a liking to me and would follow me around. I would be trying to politely end the conversation or move to another area, and he would keep coming. I would have a manager talking to me over our walkies, and he would still be conversing with me. I couldn't flat-out tell him off because he was a customer, but I was terrified of getting in trouble for shirking my actual work, and really did not want to be talking to him.

I did sneak away any time I saw him in the store to avoid interacting with him, though. I think he specifically looked for me when he came.