From my experience 98% of your stoner friends aren't friends, you just share a common interest and that's it.
Also when you get older people like to think it should still be like it was when you were in your late teen years/early 20s, lots of people, lots of party.
But the truth is people move on. They get a new job, a partner, kids, a house etc. Hanging out, playing games and getting drunk might still be fun but there are now other things wich are (more) important to them (us).
Plus: real friends are very rare. Often it's just a group of buddys-not friends. In german you can differentiate between Freund (friend), Kumpel (buddy?) and Bekannter (person you know-sometimes well and for a long time).