That's true it is just a factor. You should never stay in one place because you like it, there are other factors that come into play such as the career growth, the challenges of moving cities/continents, meeting a partner, marrying partner, starting a family, buying a house etc. These play into those emotions too, and it's the general emotion of leaving that safety net, the place of comfort.

It's tough making those jumps, but the reward is worth it, especially if you have someone with you being your cheerleader helping you through that transition period.

TL;dr totally agree it is one factor in the decision to leave. Don't let it be the only factor to stay.