I don't know what it is for other people, but for me I have a difficult time comprehending what someone is saying and why when I can't see their face. I also hate that you can't tell when someone is done speaking or about to speak. The anxiety part isn't so much during the phone call, but when I replay all the stupid stuff I said and awkward moments for hours after the phone call is over.
Luckily I have Tricare. I can get referrals, new prescriptions, have questions answered etc using an app that has a messaging system that messages my PCM clinic. I believe it's also used by the Canadian healthcare system? It's great for me because I avoid phone calls and office visits most of the time and it's great for the clinic by reducing silly appointments.