From my experience:
Keeping app switcher empty when not using the phone for a while is definitively not decreasing performance, battery charge or giving worse usage experience. By contrary, having memory resources available when launching an app is actually more beneficial on my normal usage. “Free memory is wasted memory” and “system is doing more better job than user to offer better experience” are again just mantras many are using. Truth is devices intended to run good graphics apps and Internet browsing are just severely crippled on just 512MB memory. More that the latest iOS, even with dumbed down UI, is using memory resources over 80% with no apps running and also 1GB RAM is not enough. iOS 6 was quite smooth even on devices 512MB RAM, keeping about 40% memory resources available but still crashing browsers with more than 2-3 tabs open. iOS 7,8 just destroyed those devices and even newer devices are struggling. Also apps taking longer to open if they are not in the switcher, if you open couple of apps successively, iOS will force clear apps in the background cache to make room for newer app and same app will take little longer to open. There are just not enough hardware resources. Got better battery usage between charges from not keeping apps in switcher. But if you keep opening same apps very often it might not even make sense to keep removing them from the switcher.
There are also some articles about keeping the app switcher clean to increase performance on your device, but still many are falling for the opposite. One is this: