Rethink your attributes and rethink them again. Nothing bogs down a store and adds overhead like too unneeded attributes and a attribute set mess. Doing it right the first time avoids headaches on down the road.

320 products, 50 attributes really isnt a whole lot. The default import functions of Magento should be adequate. Manually setting up the attributes and attribute sets shouldnt take but a morning either. By the time you install some bloated extension or figure out how to do it server side you could have it done from the admin CMS.

Second question; is the sub-categories imported as attributes?

Categories dont have attributes. Products do.

To turn on layered navigation (attributes filtering) you simply set the category as Anchor.

Final question; for the products that have attributes not applicable should I really have a separate attribute set OR is there a method to hide the empty attribute from the product spec table?

You can make edits to attribute.phtml to hide empty attribute fields or just use two lines of jQuery and hide anything thats "No" or "N/A" in the attribute field display table.