I should start by saying that if someone wants to smoke weed that is entirely up to them and I can accept that. I do not, however, wish to date them (with a view to a long-term relationship). There are a few causes for this:
While it is diminishingly the case, weed is still illegal in many states, and as an immigrant here there is no way I'm jeopardising my visa (and my career) for it. While this scenario is low probability, the consequences are sufficiently dire that it's a risk I'm not willing to take.
On a related note, if my partner or I ever need to return to my home country (even just to visit my family), I don't want to deal with them being refused a visa because of a previous drugs possession charge. Again, unlikely. But my family takes precedence.
I don't enjoy being around people who are smoking weed. I don't even enjoy being around drunk people (I don't drink myself). The idea of willingly impairing (in a manner of speaking, or altering if you prefer) my brain function is not only an antithesis to everything I hold dear, but is just distinctly not enjoyable in any way shape or form. That's entirely personal preference - I'm also not going to date someone who likes to go for long boozy brunches, or who enjoys getting 'wicked drunk' (to quote an OkCupid question) as while there's nothing inherently wrong with indulging in that ... it's just not what I derive any joy from.
On a final, frivolous-by-comparison note - I hate the smell. Nay, I abhor the smell of weed, the way it sticks to clothes and lingers around. And I'm not signing up to deal with that.
So I don't think people who smoke weed are bad or wrong or insert negative descriptor here. But they're not for me. Just as other people have and will choose not to date me for not drinking.
Or in other words, I can respect someone's right to do something, but not wish to be in any way related to it myself ('Murica).
(And for the record, since most people wonder when they read something like the above, I'm neither conservative, nor religious).