It sounds like you want to get to the core of working wood, which is a little different but just as satisfying as modern woodworking. I'll share my experience: Some time in my early teens I discovered the PBS program the Woodwright Shop with Roy Underhill. Basically, a show about historical woodworking, no power tools, working from a felled tree and turning it into a chair. I was super excited, I didn't need a woodshop just an axe and a few old tools (turns out hand tools are just as expensive as power tools). Hit the flea markets and got myself an axe, hatchet, bench plane, drawknife, and a couple of hand saws. The first wave of tools. These would lead to my first few projects like replacement tool handles and mallets and walking sticks, all done in my tiny basement workroom. From then on I only bought a tool when I needed it for a specific project. All my wood was cut in my backyard (which lead to learning tree and wood ID). In those few years I ended up learning more about how wood actually works than 80% of modern woodworkers.
Around the same time I came across this website which showed me a whole other side of the hobby not many people know about. This was like the most hardcore thing you could put a piece of wood through, you had to know how wood dried and grew and seasoned, you had to know how it moved and bent, ended up learning every species had completely unique properties. And to top it all off, it didn't take very many tools. I ended up spending 50 dollars for a rasp, file set, clamps, glue. Also cut down some persimmon trees for future bows. Keep in mind, my experience with archery was a little 15 pound bear bow my uncle gave me when I was 7.
Bow making was when my skills really took off and it became an art http://imgur.com/kTbZqJh
My point is, there's a facet of this hobby for everyone. Don't let people tell you it's an expensive hobby because if you just want to work with wood you can. Whittling with a pen knife, restoring tools, or bow making doesn't require much.... Nakashima style furniture might, but cross that bridge when you come to it. Hopefully this post will sow a seed and help someone in the "but I don't have a shop" mindset.
Oh and shameless plug for /r/Bowyer