I don't think the world (nor I) will be telling you not to. By all means, go nuts. It's just that based on your question I can guess that you're not that experienced with Linux and will have quite a learning curve ahead of you. I just meant that running Windows on the host would be the easy way to accomplish what you're trying to do. If you want to put in the time and effort, kudos to you.

The first thing you should do, IMO, is to verify that you have VT-d/AMD-Vi support and that your GPU is in a separate IOMMU group (and that they implemented it properly which is not always the case in consumer hardware). Oh, and you'll lose the ability to use your GPU on the host so you'll need to plan for that.

How much resources you save is really depending on what you have installed and what's running in the background.