Are there leftover files? Yes. There are files in your WTF\Account\Your-Account\ that store the settings for each addon you run. If you delete the addon, those files remain. If you uninstall from the Curse Client (and presumably other clients), those files are removed.

Does WoW explicitly read settings files from addons you have disabled or deleted? Very unlikely. Would reading those files incur a performance hit versus not having those files at all? Technically yes, but mostly in loading up. If WoW iterates all files in a directory for some reason, it might take a few milliseconds longer because of some extra files in the folder.

Will that create a noticeable drop in framerate? No. Will that create a theoretical fraction of a digit drop in framerate? Maybe? WoW isn't bottlenecked by disk performance except when starting up... but even then, it shouldn't be reading those files since it wouldn't gain anything by doing so.

If you're as obsessive about FPS as you say, maybe go get the proper tools to test average FPS over time on your setup before and after removing those files and decide? IMO, it's certainly not enough to warrant burning the freaking house down and rebuild every time you want to stop using an addon, but I'm not you.