You're 24 years old, and have only been with him for 3 months, just dump him. It doesn't matter if they are only friends, you are uncomfortable, that's a good enough reason to break up.

Dating is like a job interview. You date to figure out if he's a good fit. If this friendship is making you uncomfortable, then he's not a good fit. There are other candidates out there.

You guys are not compatible. It doesn't make you or him a bad person. Break up with him now. Don't waste years of your life stressing or being anxious about his friend. Go interview other candidates. You are trying to find someone for a lifetime position - being your partner. This is the most important decision you will make, don't settle for anything less than a perfect fit.