I'm an animator so i think about this stuff a lot. Here's my take.
Animation is not about something happening. It's about the idea of something happening. No matter how good we do our jobs, how perfectly it's rendered, how beautifully it's animated... it's not real. And the audience knows it. In the back of their minds at all times they know what they're looking at is not real. The only way that it works for them, they only way they can believe it is in how much of themselves they can put into that character. Toy Story is not about Buzz and Woody. It's about the idea of seeing your best friend hanging out with someone else. About the idea of being replaced. That goes for all the great Pixar films. Incredibles is about not living up to what you know you can be. About the mother who's being pulled in 10 different directions at once. About the teenager who just wants to disappear. About the kid who's got so much energy he's just bursting at the seams. Finding Nemo isn't about those fish. It's about that shitty thing you said to your parents that one time. About your best friend who can drive you up a wall you know they'll be there for you. These are all basic human experiences that everyone can relate to in some way.
The magic of hand keyed or hand drawn animation is because we allow the audience to interpret what we're telling them. It's not perfect or literal. We give them the idea and they see it in their own way.
It's a classic scene in so many ways but by today's standards it's poorly done. Spacing is kinda wonky in places. Volumes change. But that doesn't matter because it's not about that thing happening. It's about the idea. And because of the imperfectness every single person who watches it sees it in their own way. They fill in the blanks themselves and make it their own.
Again a classic scene but would never pass muster in any animation studio today. But again it's not about the fucking animation. It does it's job. It gives you the idea. You fill it in yourself.
So herein lies the problem with motion capture. It's not about ideas anymore. It's a literal interpretation of movement. There's no room for the audience. Instead of describing something happening it's telling you literally what's happening. So it becomes about that thing happening there on screen. It's the same with live action. It's about that thing happening there at that time and not about how you see it.
Avatar works. I think it's an amazing film. And the reason for that is because it doesn't treat itself like an animated film. You believe those things are happening there and it's entertaining as shit. Main reason for that to me is camerawork. There's not one camera in Avatar that couldn't exist in real life. Because of that it never betrays the fact that what you're seeing doesn't exist.
Look at this As opposed to this Or this
In the second two you can see the camerawork betraying the reality the motion capture is trying to sell. So you end up with a bunch of audience members being told something is happening literally by the animation but being undercut at every turn by the camerawork. The two end up fighting in your head and you just end up grossed out.
So I guess my main point is motion capture can work. But you can't think of it as animation. The audience won't accept it. Think of it as a reality and you've got a chance. Now if you want me to talk about games that use motion capture that's a whole other thing :)