It's not like Toronto has this huge population boom gobbling up all the housing.

It gains an average of about 25,000 people per year. That is a lot of housing that needs to be constantly built to keep up with the needs of the population. It may not be a boom, but you don't need a boom to put upward pressure on the market.

Even being just one home short of housing everyone is enough to send the price skyrocketing.

The price moves are because of low interest rates causing people to buy housing they can't afford

If that were the case, we'd see price rising everywhere. People across the entire country have access to mortgage lenders too... Yet, the average home price where I live held completely steady (in nominal dollars, no less!) from 2008 to 2015. It has only recently seen a small increase, and is mostly a correction against not keeping up with inflation in those previous years.