Civilian here with a little bit of knowledge. If you're referring to Special Forces as in the Army Green Berets, then I won't be much help in terms of specifics, but I can give the general idea of why a solider from the Special Operational Forces (SOF) is that much better than your average Infantry solider.
Selection: Any Special Operations in the U.S. Military has an extremely rigorous selection phase such as BUD/S for Navy SEALs, SFQC for Army Green Berets, RASP for Army Rangers, etc. The idea of these selections is to weed out those who are not mentally & physically equipped to handle the stress of combat. BUD/S for an example, is a 6-month long selection course that every SEAL must go through. It has an attrition rate of 80%+, and we are talking about a group of already-extremely-fit guys. TLDR: Only the ones who want to be there make it.
Training: All of the Special Operations Forces have intense and extensive training. And I mean extensive training. They are taught to fight in different environments, manipulate different weapon systems, and not to mention, fight harder and faster than anyone else around. There's a reason why they are referred to as high-speed sometimes. Obviously, different SOF Units will have their speciality. Air Force Para-Rescue Jumpers specialize in personnel recovery, Green Berets specialize in unconventional warfare, SEALs specialize in underwater combat-demolition, etc. TLDR: Ridiculous time & effort for training.
Funding: Self-explanatory. They get all of the cool gear, from high-tech NODS (Night Vision Goggles) down to whatever chest rig they want to run. Not to mention, SOF Units will most likely be able to shoot as much as they want, which means train as much as they want. And practice, makes almost-perfect.
TLDR: Lots of money. Lots.
War: A lot of responsibility is placed on SOF Units during wartime. This means a lot more combat-experienced men within their unit/group/platoon, and this will transfer on to incoming members. Like I said earlier, can't speak for Green Berets, but for SEALs, during SQT (SEAL Qualification Training), SEALs from a recent war-zone will pass down knowledge to the newly trained SEALs about particular ways to fight, move, etc. This entails a much more refined warrior for a particular environment and so on.
TLDR: They are the Spartans of our time.
TLDR of the Whole Thing: Give a bunch of Olympian athletes guns and gear. Train them until they eat nails for breakfast. Send them to war.