One major aspect we overlook now is that most famous people in the past (up to ~1900s) were also wealthy enough to devote their time to their passion.

That's not quite true. If you look at Haydn for instance, he came from a family of peasants. His father had purchased a harp-like string instrument and taught himself to knock out a tune on it (basically what we would think of today as a folk musician).

His whole family was very musical, but uneducated. They had a lot of free time to devote to music making since craftsmen and peasants could largely dictate their own working hours in those days, and as the historian Eric Hobsbawm points out, peasants would generally only work for around 150 days of the year.

Obviously he had a lot of natural talent too, and he was also lucky enough that one of his relatives was a choirmaster who was able to educate him. But if you look at the big picture, many poor people had a lot of free time in those days - their children generally didn't go to school, so they could devote themselves full-time to music making.

And after Mozart, it became kind of a get-rich-quick scheme for poor families to try to force their children to become musical prodigies in order to supplement the family income. This was partly what happened with Beethoven, but also other composers like Brahms, whose early life certainly wasn't as glamorous as the young Mozart's. Instead of performing to wealthy patrons, he was playing the piano in a whorehouse at age 11 - his destitute family were so desperate for money that they would push their son to those lengths just to put food on the table.