The thing about Wordpress is that it's a catch-all system. It can do anything pretty decent, but nothing the absolute best.

If you have a very limited scope of what you want to do, then building your own system can in fact be better than Wordpress. But the problem comes months down the road when you want to add another function to your site and realize that your system has to be totally revamped or requires tons of work to add that function, whereas Wordpress could have added it with a free plugin in a few clicks.

Overall I would love to have my own system to maximize stuff like page load speed, but it just isn't worth it right now. Maybe someday when I am making a nice fat income from my sites, I will build my own system and it will be fun and worth it. But not today.