It's nonsense and it comes from people that don't know any better. I can give you both anecdotes and (uncited) stats. By uncited I mean it was released as a report last year and I don't remember where to find it again.

Statistically speaking UofT and Ryerson have more security problems than York (something like 10 per month York vs 22 or more per month at the others), but they just sign it off as downtown crime because it can be dismissed that way. On the other hand, anything that happens in the "York University Heights" area is usually chalked up to York. Basically there is a lot of inflation of crime at York because of how the region is named and how the media can get a better story out of it.

Anecdotally, if you just practice smart and safe measures you should not run into trouble. This mainly means don't be somewhere unpopulated and alone late at night. This doesn't mean doing so is dangerous, just that it opens you up to being more vulnerable. Yes, there have been instances on and off campus of people being robbed. Usually these occurrences are off campus and sometimes in the village residence, and when it is darker out. In some cases it has happened in the central area, which would be the most concerning, or on the outskirts of campus (this would be the main frequent location I think).

Your mileage will vary, but I can definitely say the media exaggerates and inflates what happens at York, and that if your sources of how dangerous it is are people that have never actually been in an incident but are just complaining about the media reports and security bulletins, chances are they are just expressing their bias/fear/concern and communicating it as the reality.

I would recommend against living off campus and driving if safety is your biggest concern. I'd suggest trying out residence first year and see what you think. That way, you would have easy access to GoSafe, which are paid people that basically escort you across campus.

Also, you can google yorku security reports; they publish weekly reports of all incidences on campus that they are called for, and decide for yourself what you think.