With Android and iOS devices you have to manually prompt the device to begin voice recognition. These smart tv's are trying to recognize commands amidst all the other conversation and ambient noise in a house at all times without being prompted . It's going to be a much more complicated system. /u/rotirahn explained this already
EDIT: I want to clarify my point here. Let's say you bought a voice controlled light switch because you think it makes your life easier. If many times during the day you would say "lights on" and the the light didn't switch on what would you think of that product? You would think it is a piece of shit. That would miss its main purpose which is to turn the light on.
To prevent this, the light switch should not miss the voice command that it is set to start working. But how is it even possible to not miss it? Should it have a button to activate listening mode first? No because it's purpose is to replace buttons. Should it have a keyword to activate broader voice commands? No because it's basically same, a keyword is still a command. The device has no option but to listen to all conversations.
And also from the Samsung website
If you do not enable Voice Recognition, you will not be able to use interactive voice recognition features, although you may be able to control your TV using certain predefined voice commands. While Samsung will not collect your spoken word, Samsung may still collect associated texts and other usage data so that we can evaluate the performance of the feature and improve it.
You may disable Voice Recognition data collection at any time by visiting the “settings” menu. However, this may prevent you from using all of the Voice Recognition features.
No one is making you use voice recognition. If you bought one of these you're essentially a beta tester who bought into the hype of smart technology, which has a documented history of privacy invasion. This is Xbox One all over again, except with Xbox One it was a legitimate issue because at the inception of the smart features there was no option to use the console without them.
There's a simple solution to not having your privacy invaded by products like this: don't buy them, or turn the features off.