I wouldn't call TouchWiz a small detail. That's... you deal with that virtually every time you touch the phone.
The biggest issue I remember with Samsung is how much excess there is on that phone. I pick up my brother's phone, right now, I'll see - some dumb fucking quicklaunch sidebar Samsung thought the phone needed, an update to some Samsung services thing, a request to login or make a Samsung account, and notifications from two different email clients, Google's and Samsung's.
The other issue people have is, they're the biggest name in Android, for what. TouchWiz itself has a goofy dated look, if you ask most Android users, and it looks awkward when you have an app drawer with two different design styles (TouchWiz and Android) depending on whether the app is stock or downloaded. Samsung phones feel cheap, or at least people say they do. The iPhone knockoff lawsuit didn't help. So Android fanatics look at this, and they look at how Samsung gets to be the face of Android and the default iPhone alternative and the biggest name even though when the HTC One M7 came out it was beautifully designed and nicer than any Samsung and definitely closer to iPhone quality, and the Moto X had the active display and a modern software design while Samsung makes phones with icons that could've come off an LG Chocolate Touch.
I don't follow Samsung too closely. It looks like they've changed a bit. But for a long time, it's seemed their app designs look toyish and prepaid, phones look sorta cheap (that weird oval home button... really?) their high-end phones are filled with bloatware, they've got this issue where they're trying to make an OS on top of another OS, so the phones feel half-Android, half-Samsung, their lineup is confusing and has too many members... and they get to be the biggest name. That's the real issue. The hardcore, Nexus-buying, stock-android-loving, Cyanogenmod booting Android fanatics are pissed that Samsung, with all their perceived lack of professionalism, is the king while the prince of Android, the new Moto X or the Sony Camaro ZL1 or OnePlus or whatever they're fucking circlejerking about this week stays relatively unknown.