yes, you are justified in feeling uneasy. I was the husband in the same situation, a woman at work got along with me really well and we talked a lot. She was coming to my house for visits and my wife was getting uncomfortable. I was naive, I liked spending time with her and made the work day fun. I had a few arguments with my wife about it and she demanded I stop talking to her. I started going for a drink with her on fridays, eventually started going for a drink at her house. Eventually, I saw that my wife was right. She came on to me hard one night and we kissed and masturbated each other. I felt horrible and we don’t see each other anymore, I broke off our friendship and she quit her job. But I should have listened to my wife, I was just do naive I liked the attention