These are pretty much the reasons that I couldn't work remotely at this point in my life. I think it only makes sense if you have a family (SO/kids) or friends that you are going to interact with every day regardless of work. Probably a better situation for someone closer to retirement age.

I had a job where I worked for a very small company. While I worked in the office, there was essentially no one around all day long. I would basically say hi to a secretary on my way in and then work alone for 8 hours. I actually started to get annoyed when I wasn't included in meetings. It felt like all they wanted me to do was sit in front of a computer. I ended getting pretty depressed. The thing that saved me was that I moved in with a friend, having a roommate made a big difference.