I have not lived in Madison, however I have looked up photos on google images before...
And I totally realize how little that qualifies me to speak of life in that town, but what I can say, is that it looks like a pretty, small city in the middle of an otherwise rural, northern area.
I could use that same sentence to describe Burlingon, maybe replace pretty with gorgeous..
Therefor, as what I'm told by everyone is natural born dickhead status for being from NJ around the Philly metro, and currently living in Philly, I would halt judgment and assume Madison might be as cool as Burlington.
Either way....
Burlington is absolutely awesome. The people are nice, the food is good, the skiing is good, the outdoor recreation, scenery, and beer is amazing.
You should move to new places even if your home is different. Unless you hear the place is awful, but Burlington is not. I've never experienced more accessible natural areas while still having an active nightlife. I'm not even talking half hour commutes.. Within a fifteen or twenty minute walk along the waterfront bike path, you can find beaches and parks with very secluded and natural feels to them.
Favs about Burlington:
The bike path is absolutely gorgeous.
This plays into the last thing I just said, but the colchester causeway at the northern end of said bike path is incredible.
Culture is very laid back, but still having an emphasis on making the most of life. Basically the people are very relaxed and leisure oriented, but not necessarily lazy.
Food is great. It may seem like a small town, but the restaurants overall are decent, and in a larger city, you'd be paying 30% more for the same quality of food.
Church st marketplace is as nice of a center of town as you can get.
Five or so beaches right within city limited, plus loads of parks.
Awesome beer, but that's really any of the larger towns / cities in VT.
This is a pro that depends on a con, but nothing feels better than the first 65 degree day after having a six month winter that included many nights of windchill under -20F.
Cons:
If you're above 25 years old, the college and shortly post college nightlife downtown may seem overwhelming.
The city gets more and more crowded every year.
There's no US major city within two hours. It's like 2.5 hrs driving to Boston if I recall.